Thanks for a thorough and well organized post. This is a good topic. We have a simpler stack but still found challenges with Serverless Framework, especially including cases where we made structural changes -- the underlying CloudFormation updates often broke for one reason or other. We also ran into limitations where we had to integrate basic CF code, which was better than the alternative, but kind of seemed weak. I found that in the end, it is pretty good, but just another thing to know, learn, keep up to date with, deal with when it breaks, and so on. Dealing with any of this is just a battle with complexity, and for me, Serverless is a layer that seems heavier than I would prefer.

My experience with pretty much everything AWS does, however, is that it's a huge surface area to grasp. All their products are "one size fits all" so attempt to be complete, at the expense of having to filter out the numerous use-cases that just don't apply for me. I also have found that every service is different, so just because you know service X doesn't mean you're that much closer to understanding service Y. AWS Glue Jobs (now AWS Apache Spark) and related tech like Athena, Data Catalog, Workflows, S3, and all the rest are a prime example. I spent a solid chunk of time just getting a functional system in place a year or so ago.

I haven't looked that carefully at CDK -- I am curious if it's a cleaner and more consistent developer experience than the other services AWS offers.

I also spent a couple hours looking at what it would take to do all of this with Terraform, which we use for other infra stuff. It was pretty good overall, but (as you describe with CDK) involves rethinking how the app is structured and deployed. Too many moving parts for Terraform to do it all.