hey guys, not sure if this is a right place for this, but we just had a major prod issue with prisma, and we wanted to see if someone can help us figure out a root cause. Basically, we have 2 prod prisma servers running in Docker, both pointing to a same MySQL server in Azure, but using different data and migration databases (one prisma server for internal app, and another for external app). Today, out of the blue, both prisma containers lost connection to the server, we had restarted them, one came up, but another kept failing to obtain an exclusive agent lock (according to logs). It was very strange, if we stop one of the servers (at random), another obtains the lock just fine, but not both at the same time (which had worked previously just fine for an entire year that we had been running prisma in production). What is even stranger, after about an hour, the problem cleared out and both server can now get a lock to the same DB server as before. I guess what we are trying to figure out if pointing 2 different prisma servers to the same DB server (but not the same DB) is even a valid strategy, even though it has worked for us before, or is there a specific setting / process that we should have followed? Any info that you can provide on this issue will be highly appreciated. Thank you very much!

Hi, Are you using two FULL prisma servers?

yes

That will cause trouble, if you want to horizontally scale it you need the prisma-prod image

By default it should not be possible that 2 Prisma servers get a lock on the same db server. The lock is there to ensure that only one Prisma server is ever migrating the underlying database. It should be possible if the configuration is done correctly though. Can you clarify the following? Please share the configurations (without passwords)

both pointing to a same MySQL server in Azure, but using different data and migration databases

marcus, here is the config for first prisma:

What about using prisma workspaces like /project1/default and /project2/default in the same prisma instance?

I wouldn’t say it is a no go but we haven’t tested this ourselves. I looked into our code for MySQL and is calling

SELECT GET_LOCK('deploy_privileges', -1);

when it starts. Our understanding is that it is server wide and therefore no 2 Prisma servers can be started on the same database.