How many of you are and aren't running your production servers on VMs etc that let you easily replace or roll back the whole thing if it gets messed up, say by a bad system upgrade? What's that look like? How easily could you do that? How long would production be down? Have you tested actually doing that? Thanks for any thoughts.

the modern way is all docker / k8s

We’re not quite to the Docker stage yet. Our production is on a VMWare vSphere VM that is snapshot every 30 minutes. I’ve only needed to roll back once. Running on Ubuntu Server 22. Downtime was about 20 minutes. I’m looking now at LXC containers, as I’m not yet docker-fluent.

Modernize or Die as I hear it lol. The ancient ways of physical boxes being back up to tapes is 10+ years old or more. VMs / Docker etc are the approach forward. Could have you back up in minutes.

All our servers are actually VMs already, managed by our systems folks. However, we're not really set up to, for instance, have a clone of production already running when we're doing an OS or other major update, so all we have to do in case of problems is point the firewall at it. Of course it's not that simple, I'm just trying to suss out what we could do to be more robust against unexpected trouble.

do you have a load balancer?

Yea create a hot spare to shift your instance over to while performing maintenance on OS etc.

@Peter Hoopes ---- um... snapshotted every 30 minutes? Do you files change that often? I hope you aren't doing that with a live DB on that system....

Data doesn’t change too often. Disk array snapshots eight before VMWare does. Local MySQL db on server… seems to be ok though that’s a good point. I didn’t do the VMWare or drive array setup. I have. DB master/slave in place so I’m ok if the DB felt unwell…

Particularly, if you do drive snapshots, you are very ripe for data corruption or data loss on MySQL. MyISAM was fine with it because it hit the disk on every write, but InnoDB is very lazy in writing data. Meaning, you could very easily get into situations where the snapshot happens after one table is written and the other is still in memory -- or worse, index are incomplete after the data is dumped from memory.

All good points. Worth a reexamination.

DB data isn't part of what I mean to be talking about here. Our DBs are on their own servers. We'd stop everything, back them up and if needed restore them separately. I'm only talking about the file system. File system does get backed up nightly I think, but I'm thinking about another mechanism, to specifically back up the directories needed right before some major shuffle. The hope is that in case of major upset, we could restore both the DBs via their own backups, and the file system via some other mechanism.

app data or app code?

App code.

separate the filesystem from the vm, attach them as a volume?

Or write a basic Shell script to copy and send them off to S3