wonderful-quill-11255
04/04/2023, 5:45 PMwonderful-quill-11255
04/04/2023, 5:45 PM2023-04-04 17:11:08,877 [INFO] [__main__] Running /config/staging/glue-rest2.yml.
2023-04-04 17:11:09,731 [INFO] [datahub.ingestion.run.pipeline] Sink configured successfully. DataHubRestEmitter: configured to talk to https://our-gms-host:443
/opt/python/3.9/lib/python3.9/site-packages/datahub/ingestion/source/aws/glue.py:292: ConfigurationWarning: env is deprecated and will be removed in a future release. Please use platform_instance instead.
config = GlueSourceConfig.parse_obj(config_dict)
2023-04-04 17:11:09,812 [INFO] [botocore.credentials] Found credentials in environment variables.
2023-04-04 17:11:09,868 [INFO] [botocore.credentials] Found credentials in environment variables.
2023-04-04 17:11:10,007 [INFO] [datahub.ingestion.run.pipeline] Source configured successfully.
2023-04-04 17:13:36,096 [ERROR] [datahub.ingestion.run.pipeline] failed to write record with workunit urn:li:dataset:(urn:li:dataPlatform:glue,dummy_database.dummy_table,DEV)-subTypes with ('Unable to emit metadata to DataHub GMS', {'exceptionClass': 'com.linkedin.restli.server.RestLiServiceException', 'stackTrace': 'com.linkedin.restli.server.RestLiServiceException [HTTP Status:500]: javax.persistence.PersistenceException: Error when batch flush on sql: update metadata_aspect_v2 set metadata=?, createdOn=?, createdBy=?, createdFor=?, systemmetadata=? where urn=? and aspect=? and version=?\n\tat com.linkedin.metadata.restli.RestliUtil.toTask(RestliUtil.java:42)\n\tat com.linkedin.metadata.restli.RestliUtil.toTask(RestliUtil.java:50)', 'message': 'javax.persistence.PersistenceException: Error when batch flush on sql: update metadata_aspect_v2 set metadata=?, createdOn=?, createdBy=?, createdFor=?, systemmetadata=? where urn=? and aspect=? and v', 'status': 500, 'id': 'urn:li:dataset:(urn:li:dataPlatform:glue,dummy_database.dummy_table,DEV)'}) and info {'exceptionClass': 'com.linkedin.restli.server.RestLiServiceException', 'stackTrace': 'com.linkedin.restli.server.RestLiServiceException [HTTP Status:500]: javax.persistence.PersistenceException: Error when batch flush on sql: update metadata_aspect_v2 set metadata=?, createdOn=?, createdBy=?, createdFor=?, systemmetadata=? where urn=? and aspect=? and version=?\n\tat com.linkedin.metadata.restli.RestliUtil.toTask(RestliUtil.java:42)\n\tat com.linkedin.metadata.restli.RestliUtil.toTask(RestliUtil.java:50)', 'message': 'javax.persistence.PersistenceException: Error when batch flush on sql: update metadata_aspect_v2 set metadata=?, createdOn=?, createdBy=?, createdFor=?, systemmetadata=? where urn=? and aspect=? and v', 'status': 500, 'id': 'urn:li:dataset:(urn:li:dataPlatform:glue,dummy_database.dummy_table,DEV)'}wonderful-quill-11255
04/04/2023, 5:51 PMwonderful-quill-11255
04/04/2023, 5:52 PMwonderful-quill-11255
04/04/2023, 5:52 PMwonderful-quill-11255
04/05/2023, 8:16 AMwonderful-quill-11255
04/05/2023, 10:35 AMwonderful-quill-11255
04/05/2023, 11:40 AMbright-spring-66365
04/06/2023, 9:55 AMloud-island-88694
gray-shoe-75895
04/07/2023, 7:05 PMbrainy-tent-14503
04/07/2023, 11:24 PMREAD COMMITTED to something else?wonderful-quill-11255
04/10/2023, 8:38 AMREAD COMMITTED.brainy-tent-14503
04/11/2023, 3:54 PMCaused by: org.postgresql.util.PSQLException: ERROR: could not serialize access due to concurrent update should not be possible if postgres is configured for READ COMMITTED is my understanding. Can you double check your AWS configuration?wonderful-quill-11255
04/11/2023, 6:05 PMaloof-gpu-11378
04/11/2023, 6:43 PMrich-pager-68736
04/14/2023, 7:55 AMaloof-gpu-11378
04/17/2023, 8:26 PMpostgresql there is an issue with parallel transactions which is not present with mysql , A change in the default max_threads from 1 to 15 in the default options as part of v0.10.1 is from this change. In order to workaround this, please update the ingestion recipes back to max_threads: 1 . Additionally a note was added to the latest release notes about this.wonderful-quill-11255
04/18/2023, 7:36 AMwonderful-quill-11255
04/19/2023, 8:33 AMfancy-crayon-39356
05/29/2023, 4:19 PMv0.10.1. Currently I've already upgrade to v0.10.3 and still having the same issue. As it was already commented here, my backend DB is aurora postgres as well.
I've tried to set on the recipe sink.config.max_threads: 1 but it didn't help 😕 This issue happens mostly in our snowflake sinks. We are running GMS with 3 replicas (in k8s), and also running standalone MAE and MCE consumers (with 1 replica each)
@brainy-tent-14503 any suggestions here? Help would be appreciated 🙏 Thanks!fancy-crayon-39356
05/30/2023, 8:26 AMDetail: Key (urn, aspect, version)=(urn:li:assertion:eff38e71b70cf7871ce86ff3e9f8faf9, assertionInfo, 3223) already exists. Call getNextException to see other errors in the batch.
ERROR: duplicate key value violates unique constraint "pk_metadata_aspect_v2"
Caused by: io.ebean.DuplicateKeyException: Error when batch flush on sql: insert into metadata_aspect_v2 (urn, aspect, version, metadata, createdOn, createdBy, createdFor, systemmetadata) values (?,?,?,?,?,?,?,?)
It seems related to the same underlying problem of concurrent operations on the backend dbgray-byte-33609
05/30/2023, 8:37 AM2023-05-30 08:24:47,532 [qtp522764626-219] INFO c.l.m.filter.RestliLoggingFilter:55 - POST /aspects?action=ingestProposal - ingestProposal - 500 - 165ms
2023-05-30 08:24:47,533 [qtp522764626-219] ERROR c.l.m.filter.RestliLoggingFilter:38 - Rest.li error:
com.linkedin.restli.server.RestLiServiceException: javax.persistence.PersistenceException: Error when batch flush on sql: update metadata_aspect_v2 set metadata=?, createdOn=?, createdBy=?, createdFor=?, systemmetadata=? where urn=? and aspect=? and version=?
PostgreSQL Logs:
2023-05-30 11:24:17 +03 [4610] user=***,db=***,app=PostgreSQL JDBC Driver,client=[local] ERROR: could not serialize access due to concurrent update
2023-05-30 11:24:17 +03 [4610] user=***,db=***,app=PostgreSQL JDBC Driver,client=[local] STATEMENT: update metadata_aspect_v2 set metadata=$1, createdOn=$2, createdBy=$3, createdFor=$4, systemmetadata=$5 where urn=$6 and aspect=$7 and version=$8
CC: @glamorous-wire-83850glamorous-wire-83850
06/12/2023, 8:45 AMaloof-gpu-11378
06/13/2023, 12:45 PMshy-scooter-37097
06/17/2023, 2:08 PMshy-scooter-37097
06/22/2023, 7:21 PMbland-gigabyte-28270
06/23/2023, 1:36 AMbland-gigabyte-28270
06/23/2023, 2:11 AMCaused by: java.sql.BatchUpdateException: Batch entry 0 update metadata_aspect_v2 set metadata='{"paths":["/prod/snowflake/arene/aha"]}', createdOn='2023-06-21 06:20:14.555+00', createdBy='urn:li:corpuser:__datahub_system', createdFor=NULL, systemmetadata='{"registryVersion":"0.0.0.0-dev","runId":"34bd9821-13ef-4a97-ae48-3e2ab7a95c59","registryName":"unknownRegistry","lastObserved":1687328414542}' where urn='urn:li:dataset:(urn:li:dataPlatform:snowflake,arene.aha.kfk_aha_epic,PROD)' and aspect='browsePaths' and version=0 was aborted: ERROR: could not serialize access due to concurrent update Call getNextException to see other errors in the batch.bland-gigabyte-28270
06/23/2023, 2:13 AMmax_threads parameter?bland-gigabyte-28270
06/23/2023, 3:36 AMbland-gigabyte-28270
06/23/2023, 3:36 AMbored-gold-25249
06/23/2023, 9:59 AM{
"error": "Unable to emit metadata to DataHub GMS: javax.persistence.PersistenceException: Error when batch flush on sql: update metadata_aspect_v2 set metadata=?, createdOn=?, createdBy=?, systemmetadata=? where urn=? and aspect=? and version=?",
"info": {
"exceptionClass": "com.linkedin.restli.server.RestLiServiceException",
"message": "javax.persistence.PersistenceException: Error when batch flush on sql: update metadata_aspect_v2 set metadata=?, createdOn=?, createdBy=?, systemmetadata=? where urn=? and aspect=? and version=?",
"status": 500,
"id": "urn:li:assertion:093266a24bb02758afec92c840b49ee1"
}
}
In the gms log we can see the following (shortened) error:
Caused by: java.sql.BatchUpdateException: Batch entry 0 update metadata_aspect_v2 set metadata='...', createdOn='2023-06-23 06: 07: 24.116+00', createdBy='urn:li:corpuser:example@example.com', systemmetadata='...' where urn='urn:li:assertion:093266a24bb02758afec92c840b49ee1' and aspect='assertionInfo' and version=0 was aborted: ERROR: could not serialize access due to concurrent update Call getNextException to see other errors in the batch.
I suppose this is the same kind of error?
Thanks already for your great help here in Slack!shy-scooter-37097
06/24/2023, 7:45 AMbored-gold-25249
06/26/2023, 11:45 AM