Bash scripts can be terse and clever, or as dumb as a stump. I vote for dumb, as long as your intent is clear. Terraform wrapper script I have been refactoring a script we use to run Terraform code in our build system. One bash function scans a bunch of files from a…

I have written about my struggles with major depression, also referred to as “atypical”, or my favorite “treatment resistant” depression. A lifetime spent with treatment using normal drugs many people take to successfully treat depression left me better off, for sure, but just “OK”. OK is ok. …

I am new to AWS Glue Jobs and PySpark. I made some assumptions about how my jobs used memory. They were wrong. Glue and PySpark use almost no memory in a typical case of transformation. Sort of. In typical Python (or whatever language) programming, code like this will result in…

I am consistently impressed by my peers who understand concepts and have skills I doubt I’ll ever attain. I work with a lot of smart people at Clearbit, and damn, so often I feel like a chump. In fairness, I have been doing this since the 1980’s,so my mind may…

When I came to my company, we had a main terraform repo for building the resources needed by the 50–100 microservices we host on our Kubernetes clusters. In a subdirectory called modules we have code that defines our databases, redis, kafka, ci/cd pipelines, github repos, monitoring, namespaces and secrets for…

pglogical is a third party postgreSQL extension allowing two PostgreSQL nodes to replicate. I am working on building a zero-downtime major version upgrade tool for PostgreSQL on AWS, so wanted to be able to run tests locally. The extension is provided by 2ndQuadrant and is not part of PostgreSQL. Some…

Terraform Is Better, But Often A Nightmare We have gone through several iterations of our Terraform code now, and it’s getting better. We have moved several different implementations of modules from various sources into a central GitHub repository of modules, releases using semantic version tags, all up to TF 1.0.10 or even 1.1 and handling the several…

Terraform lets you provision some number of things — think instances in a cluster — in a single resource definition using count. This is a great feature, controlling the number of iterations, and making things like names, tags, and other attributes easy to name. But there is a degenerative use…

(Oct 2021) — in the last several months, AWS EKS has made some minor but significant changes to Managed Node Groups. It’s clear where they are headed, but also clear there are still a few issues that AWS needs to resolve. I have recently updated our Terraform code to use…