This message was deleted.

I’m sure I’ll find out soon, I’ve just upgraded the secondary and it didn’t come up as primary. I have just initiated an ns migration now in time to complete the update tonight.

Always set the node you're upgrading to "staysecondary" during the firmware upgrade process as a safeguard against split-brain scenarios. I've not seen reports of that happening with recent 13.1 builds; as long as you aren't making a big leap in versions then there shouldn't have been any breaking changes. If you weren't using encrypted RPC and/or were using default RPC node passwords then that may have been it.

This was a weird one. We have upgraded tons of nodes in this environment before, but never seen anything like this. Node state just said UNKNOWN and both decided to fly as primary. Note that this happened after upgrading the second node to complete the upgrade (we had already upgraded and rebooted the first one without this problem). Encrypted RPC is ON and not using default RPC node passwords. I am pretty sure this is some sort of bug introduced in 13.1-37.38. Will definitely safeguard with staysecondary after this experience, though. Yes - Sebastian is my colleague 🙂