One question might be timing these types of changes. I've seen lots of people make changes in the evening/night to minimize downtime but the problem with that is that there can be a long tail to resolution since everyone is asleep. At least first thing on a Tuesday for config changes you know you have 8 hours to work on a problem. Not sure if others have experienced this but I always preferred as a customer to know that an outage was during a time when someone could actually work on it. Obviously this could be different for other folks and I'm not sure what your internal procedures are. Overall thanks for communicating. 🙂

@jimthedev You are absolutely right. We generally avoid pushing significant changes in the evening as response time will be significantly longer. In addition we have a centralised release to production so one person always has the full context. In this particular case we introduced a latent bug two days prior to the incident. In general we roll out changes in a way that completely avoid downtime and that allows us to do changes during normal working hours.

Gotcha! Yeah not an easy way to avoid those kinds of things. Good to hear about your internal procedures as that keeps the conversation happening.