Hello there, our environment runs three GMS instances with

.Values.global.datahub_standalone_consumers_enabled: false

and a PostgreSQL database. We see the following errors occur during ingestion of meta-data:

ERROR c.l.m.filter.RestliLoggingFilter:38 - <http://Rest.li|Rest.li> error: datahub-proddatahub-datahub-gms-5876c79b74-nknp8com.linkedin.restli.server.RestLiServiceException: javax.persistence.PersistenceException: Error when batch flush on sql: update metadata_aspect_v2 set metadata=?, createdOn=?, createdBy=?, createdFor=?, systemmetadata=? where urn=? and aspect=? and version=?datahub-proddatahub-datahub-gms-5876c79b74-nknp8Caused by: javax.persistence.PersistenceException: Error when batch flush on sql: update metadata_aspect_v2 set metadata=?, createdOn=?, createdBy=?, createdFor=?, systemmetadata=? where urn=? and aspect=? and version=?datahub-proddatahub-datahub-gms-5876c79b74-nknp8Caused by: java.sql.BatchUpdateException: Batch entry 0 update metadata_aspect_v2 set metadata='{"username":"..."}', createdOn='2023-07-14 10:59:11.565+00', createdBy='urn:li:corpuser:...', createdFor=NULL, systemmetadata='{"registryVersion":"0.0.0.0-dev","runId":"powerbi-2023_07_13-15_42_06","registryName":"unknownRegistry","lastObserved":1689581244577}' where urn='urn:li:corpuser:...' and aspect='corpUserKey' and version=0 was aborted: ERROR: could not serialize access due to concurrent update  Call getNextException to see other errors in the batch.datahub-proddatahub-datahub-gms-5876c79b74-nknp8Caused by: org.postgresql.util.PSQLException: ERROR: could not serialize access due to concurrent update

I assume it has to do with multiple GMS instances getting instructions to update the meta-data and competing for the same table in the database. How can we configure the environment in a different way to be able to support heavy (millions of MCP events daily) ingestion?

Hi @acceptable-morning-73148! We believe this to be a regression in recent versions of DataHub, but are still in the process of triaging the issue. We've also noticed this on our side! We are working to find a root cause and provide a fix. @dazzling-yak-93039 Is leading the charge on our side, and will be the primary point of contact. Cheers

What version of datahub are you using?

0.10.4

I think https://github.com/datahub-project/datahub/commit/5bab604432aa8adff3105ec38897ec0262e3df30 will fix this issue! This one is going to come out with the next release. Unfortunately, the release is blocked due to the issue John mentioned above (which is actually a different issue). If you need it sooner, you can apply this commit to your deployment! Otherwise, just hang tight.