Hey, we had a little bit problems when we tried to deploy staging servers. One of our PersistentEntityActors kept crashing at the start since it’s data model had changed. I had raised the server version a few times while having the NukeOnVersionMismatch flag enabled. It still didn’t work. I found a workaround but would like to understand why this happened on staging server but on dev or local server this same situation doesn’t cause any problems. Is there some differences on how database nuking works on staging vs dev servers? Here is the error when deploying:

[all-0] [16:57:43.421 ERR TeamManager] Entity TeamManager:0000000000 (status=Starting, numPendingMessages=0) crashed due to MetaWireDataTypeMismatchDeserializationException 'Failed to deserialize TeamManagerModel.BotTeam, did you change the type?

Hi, the difference is that on staging (and production), NukeOnVersionMismatch is false by default (when not configured in the Options.*.yaml files) whereas on develop and local it's true by default. But if it has been configured in the appropriate yaml files, then the default should not make a difference. In which yaml file(s) was the NukeOnVersionMismatch and MasterVersion specified? By default the staging environment uses

Options.base.yaml

as a base and then

Options.staging.yaml

for staging-specific overrides. But I notice now that our documentation has an error in that it says the default is

Options.stage.yaml

rather than

Options.staging.yaml

, so that could be a possible explanation. Additionally, the default option yaml paths can be overridden by the per-environment helm values file specified by

serverValuesFile

in

metaplay-project.yaml

.

When I had the issue, I confirmed from Staging Admin panel that the Nuke flag is enabled and Master version is lower than the current value in Options.staging.yaml. The nuking also worked after I fixed the crash issue in the TeamManagerActor.InitializePersisted function. I just passed null there instead of fetching data from database. After that nuking worked and I reverted changes and everything worked from there. I am just wondering is there some corner case where some entities are initialized before the version check, making the nuking option to not work? I tried to replicate this issue in local server but it worked as expected

The nuking should definitely happen before any entities are initialized. > Master version is lower than the current value in Options.staging.yaml Hmm, the master version should be equal to the one in Options.staging.yaml, assuming the server has been successfully deployed with the correct yaml files... So if it was lower than the one in Options.staging.yaml, that could be part of the explanation.

It was lower when I had the issue with deployment. It crashed everytime to the deserialization of the Entity, the version didn't change and the database persisted. After I got the deployment working with my workaround, then the database was nuked and the master version updated to the same value as in the .yaml file.

One thing we can sanity check is that the right options files are in use. First, you should check

metaplay-project.yaml

and see if it's referencing any `serverValuesFile`s, or helm values files, also called deployment files (yaml) that are usually in "Backend/Deployments/" folder. If yes, check those files then if they refer explicitly to any

options.*.yaml

files. If the deployment files are not set in

metaplay-project.yaml

, then defaults should be in use: in

backend/server/Config/Options.production.yaml

,

backend/server/Config/Options.staging.yaml

and

backend/server/Config/Options.dev.yaml

in addition to

Options.base.yaml

for all of them. So

metaplay-project.yaml

has

serverValuesFile

, and that file has

config:
  files:
    - "./Config/Options.base.yaml"
    - "./Config/Options.dev.yaml"

As an additional note, you can also reset a database with the cli using

metaplay database reset

. Just be careful to point it at the right environment!

I double checked those and the changes to the Options.staging.yaml started to be applied to server once I got it through the crash with my workaround. I'll keep in mind that cli database reset since we are updating to production next week with the same setup. I'll let you know if the same problem reoccurs there