For those that are one a very heavy event driven architecture, do you have any advice/tips for documentation of how various stacks, lamdba functions, etc. fit into your overall system and what does when? The company I work for is about to hire a handful of engineers as we grow and us two current devs are just thinking through onboarding and long term maintainability and explaining the whole ecosystem to new hires and reduce domain knowledge that just lives in our heads. Any tips and thoughts would be appreciated!

The mental metaphor I like to use when thinking about a system is to forget about technology and design big office building where each floor is a department with humans working. I like to look at the business and understand what departments it would make sense to have. For an ecommerce business it would make sense to have an Orders department to process incoming orders. A billing department to charge credit cards. A fulfillment department to ship out the product. Then I like to imagine a messaging system and think about what messages each department creates and cares about listening to. Each department shouldn't care too much about which other ones exist, only the messages that are available to be consumed. They shouldn't really care about the internal operations of the department either. This roughly maps to your architecture on the serverless side. Each department is a service that might have their own lambdas, databases, etc. Should be as self contained as possible and people working there should be able to work entirely on their own. Some things will go outside these, sometimes people from one department will have to call up another department to get an answer but this is minimal. When you're early the reality is all your engineers will know about all the domains but it's key to make sure the above structure is always possible. This will make it viable to have someone new join, operate just in one domain to start so they can hit the ground running and then slowly learn about the others. Documenting the message types that are available is key, they can learn about how they flow over time.

@thdxr This needs to be a blog post!

I recommend event sourcing pattern with EventBridge. The dbs should be for ease of access of data and aggregations, just a persistence layer. But your services should be built in such a way that you could use EventBridge’s event archives and replay the events in them to recreate your service even if your db is completely lost. This is a complicated and tedious pattern to implement but once you get this right you could never have any problems in adding new services or worry about data loss. Everything could be created from the events that your services process.

👍 to eventbridge, works great for us so far. Also, if you define your events with JSON schema you can leverage tools like EventBridge Atlas to have auto-generated documentation: https://eventbridge-atlas.netlify.app/ (I haven't used this yet, but it caught my eye as something we'd want to use when we get larger)