https://datahubproject.io logo
Join Slack
Powered by
Hi, ingestion tried to run twice at once why it ha...
# troubleshoot
b
Hi, ingestion tried to run twice at once why it happend? https://github.com/datahub-project/datahub/issues/7053
@gray-shoe-75895 Could we discuss about it? actually dont know how to solve it. I don't have any scheduled green launch for today. the run time are not the same.
b
Hey, Can you see any logs on GMS? what cause the 500? what caused the Failure. Can you also check please the number of action pods running?
b
@bulky-electrician-72362 just a moment
111004.997 [qtp1125736023-93487] INFO c.l.m.filter.RestliLoggingFilter:55 - POST /entities?action=ingest - ingest - 500 - 17ms
111004.998 [qtp1125736023-93487] ERROR c.l.m.filter.RestliLoggingFilter:38 - Rest.li error: com.linkedin.restli.server.RestLiServiceException: javax.persistence.PersistenceException: Error when batch flush on sql: update metadata_aspect_v2 set metadata=?, createdOn=?, createdBy=?, createdFor=?, systemmetadata=? where urn=? and aspect=? and version=? at com.linkedin.metadata.restli.RestliUtil.toTask(RestliUtil.java:42) at com.linkedin.metadata.restli.RestliUtil.toTask(RestliUtil.java:50)
we run one pod
"failures": [{"error": "Unable to emit metadata to DataHub GMS", "info": ' '{"exceptionClass": "com.linkedin.restli.server.RestLiServiceException", "stackTrace": ' '"com.linkedin.restli.server.RestLiServiceException [HTTP Status500] javax.persistence.PersistenceException: Error when ' 'batch flush on sql: update metadata_aspect_v2 set metadata=?, createdOn=?, createdBy=?, createdFor=?, systemmetadata=? where ' 'urn=? and aspect=? and version=?\\n\\tat com.linkedin.metadata.restli.RestliUtil.toTask(RestliUtil.java:42)\\n\\tat ' 'com.linkedin.metadata.restli.RestliUtil.toTask(RestliUtil.java:50)", "message": "javax.persistence.PersistenceException: ' 'Error when batch flush on sql: update metadata_aspect_v2 set metadata=?, createdOn=?, createdBy=?, createdFor=?, ' 'systemmetadata=? where urn=? and aspect=? and v", "status": 500, "id": "urnlicorpuser:e.luneva"}}, {"error": "Unable to ' 'emit metadata to DataHub GMS", "info": {"exceptionClass": "com.linkedin.restli.server.RestLiServiceException", "stackTrace": ' '"com.linkedin.restli.server.RestLiServiceException [HTTP Status422] ' 'com.linkedin.metadata.entity.validation.ValidationException: Failed to validate record with class com.linkedin.entity.Entity: ' 'ERROR :: /value/com.linkedin.metadata.snapshot.CorpUserSnapshot/aspects/1/com.linkedin.identity.GroupMembership/groups/5 :: ' '\\"Provided urn urnlicorpGroup:sccm-Amadeus Altea INV (RIT HD\\\\\\" is invalid\\n\\n\\tat ' 'com.linkedin.metadata.resources.entity.EntityResource.ingest(EntityResource.java:212)", "message": ' '"com.linkedin.metadata.entity.validation.ValidationException: Failed to validate record with class ' 'com.linkedin.entity.Entity: ERROR :: /value/com.linkedin.metadata.snapshot.CorpUserSnapshot/aspects/1/", "status": 422, "id": ' '"urnlicorpuser:d.zharikov"}}]
image.png
Don't know why ingestion runs twice at time
b
Do you run a single GMS replica or multiple? Which version are you on? Asked the team about this and it used to be a problem.
Multiple runs: This likely happens if there are multiple GMSes, but standalone consumers are not enabled.
b
single, 0.9.6.1 latest @bulky-electrician-72362
I think problem only is in schedule run and in an attempt to write the same object at the same time.
@bulky-electrician-72362 will you reopen my case or etc...?
b
Sure, can you try recreating the ingestion?
b
@bulky-electrician-72362 yep, even each new ingestion works same way. so we have no any proper schedule way to run. only manual.
b
each new ingestion works same way
do you mean every scheduled ingestion, runs twice?
b
@bulky-electrician-72362 yes!
b
even after you recreated them?
b
just recreated everything ... seems to be a normal ... hm... @bulky-electrician-72362 i will come back to you soon after tests
b
ok. please report back either way
b
@bulky-electrician-72362 it doesn't help.(((
image.png
run everything manually and it start to be green. even it was run all at same time.
the strange thing is when I recreated everything, I installed the launch to schedule after a couple of minutes and it worked well. but the scheduled launch the next day did not work.
b
Could you check the schedules in your myqsql or postgres?
are the schedules registered once or twice there?
How did you deploy the application? with helm?
b
@bulky-electrician-72362 helm chart is barely splitted and cutted in parts to be able deploy on private k8s cluster
this is postgres
of course registered once.
b
I mean the postgres database behind DH, where it stores its state. Could you connect to the datbase and query for schdeuls in it?
b
Yep, just a moment
@bulky-electrician-72362
two!
metadata_aspect_v2_202302031553_edp_datahub_p.csv
metadata_aspect_v2_202302031553_edp_datahub_p.csv
yesterday, when I recreated all the ingestions, I put them on launch after a couple of minutes and everything worked out without errors. but after i change s start time i create a second version. thats why yesterday was green but today red. I making a test now. trying to create one fresh ingestion and never edit it. I think it will run properly
yes, when i recreate again and run it by shedule without editing it runs properly
but if I edit it at least once, it will break
b
it will break
it will run twice?
do you have everything runing in one inctance, right? gms, frontend ?
b
we have two instance, one for frontend and one for action.
one instance per type (front, gms, mae, etc)
...yep it will run twice.
b
Did you set up standalone consumer related env vars properly ?
b
This is our env for GMS env: - name: ENABLE_PROMETHEUS value: 'true' - name: MCE_CONSUMER_ENABLED value: 'true' - name: MAE_CONSUMER_ENABLED value: 'true' - name: ENABLE_PROMETHEUS value: 'true' - name: ENTITY_REGISTRY_CONFIG_PATH value: /datahub/datahub-gms/resources/entity-registry.yml - name: DATAHUB_ANALYTICS_ENABLED value: 'true' - name: EBEAN_DATASOURCE_USERNAME value: >- vault:kv/kubernetes-epaas/data/p/datacatalog-prod/datahub.sql#username - name: EBEAN_DATASOURCE_PASSWORD value: >- vault:kv/kubernetes-epaas/data/p/datacatalog-prod/datahub.sql#password - name: EBEAN_DATASOURCE_HOST value: 'vault:kv/kubernetes-epaas/data/p/datacatalog-prod/datahub.sql#host' - name: EBEAN_DATASOURCE_URL value: 'vault:kv/kubernetes-epaas/data/p/datacatalog-prod/datahub.sql#url' - name: KAFKA_BOOTSTRAP_SERVER value: 'datahub-kafka-headless:9092' - name: KAFKA_SCHEMAREGISTRY_URL value: 'http://datahub-kafka-schema-schema-registry:8081' - name: ELASTICSEARCH_HOST value: elasticsearch-master - name: ELASTICSEARCH_PORT value: '9200' - name: GRAPH_SERVICE_IMPL value: elasticsearch - name: METADATA_SERVICE_AUTH_ENABLED value: 'true' - name: DATAHUB_TOKEN_SERVICE_SIGNING_KEY value: >- vault:kv/kubernetes-epaas/data/p/datacatalog-prod/datahub.tokens#signing - name: DATAHUB_SYSTEM_CLIENT_ID value: __datahub_system - name: DATAHUB_SYSTEM_CLIENT_SECRET value: >- vault:kv/kubernetes-epaas/data/p/datacatalog-prod/datahub.tokens#client - name: UI_INGESTION_ENABLED value: 'true' - name: SECRET_SERVICE_ENCRYPTION_KEY value: >- vault:kv/kubernetes-epaas/data/p/datacatalog-prod/datahub.tokens#client - name: SEARCH_SERVICE_ENABLE_CACHE value: 'false' - name: VAULT_ADDR value: 'https://secrecy.group.s7:8200' - name: VAULT_SKIP_VERIFY value: 'false' - name: VAULT_AUTH_METHOD value: jwt - name: VAULT_PATH value: kubernetes/epaas/caas31p28 - name: VAULT_ROLE value: any - name: VAULT_IGNORE_MISSING_SECRETS value: 'false' - name: VAULT_ENV_PASSTHROUGH - name: VAULT_JSON_LOG value: 'false' - name: VAULT_CLIENT_TIMEOUT value: 30s - name: VAULT_LOG_LEVEL value: info - name: VAULT_CACERT value: /vault/tls/ca.crt
for the front
- name: ENABLE_PROMETHEUS value: 'true' - name: DATAHUB_GMS_HOST value: datahub-gms-datahub-gms - name: DATAHUB_GMS_PORT value: '8080' - name: AUTH_OIDC_ENABLED value: 'true' - name: AUTH_OIDC_CLIENT_ID value: >- vault:kv/kubernetes-epaas/data/p/datacatalog-prod/datahub.wso2#client_id - name: AUTH_OIDC_CLIENT_SECRET value: >- vault:kv/kubernetes-epaas/data/p/datacatalog-prod/datahub.wso2#client_secret - name: AUTH_OIDC_DISCOVERY_URI value: >- vault:kv/kubernetes-epaas/data/p/datacatalog-prod/datahub.wso2#discovery_uri - name: AUTH_OIDC_BASE_URL value: >- vault:kv/kubernetes-epaas/data/p/datacatalog-prod/datahub.wso2#oidc_base_url - name: AUTH_OIDC_USER_NAME_CLAIM value: sub - name: DATAHUB_SECRET valueFrom: secretKeyRef: name: datahub-frontend-gms-secret key: datahub.gms.secret - name: DATAHUB_APP_VERSION value: '1.0' - name: DATAHUB_PLAY_MEM_BUFFER_SIZE value: 10MB - name: DATAHUB_ANALYTICS_ENABLED value: 'true' - name: KAFKA_BOOTSTRAP_SERVER value: 'datahub-kafka-headless:9092' - name: ELASTIC_CLIENT_HOST value: elasticsearch-master - name: ELASTIC_CLIENT_PORT value: '9200' - name: DATAHUB_TRACKING_TOPIC value: DataHubUsageEvent_v1 - name: METADATA_SERVICE_AUTH_ENABLED value: 'true' - name: DATAHUB_SYSTEM_CLIENT_ID value: __datahub_system - name: DATAHUB_SYSTEM_CLIENT_SECRET value: >- vault:kv/kubernetes-epaas/data/p/datacatalog-prod/datahub.tokens#client - name: VAULT_ADDR value: 'https://secrecy.group.s7:8200' - name: VAULT_SKIP_VERIFY value: 'false' - name: VAULT_AUTH_METHOD value: jwt - name: VAULT_PATH value: kubernetes/epaas/caas31p28 - name: VAULT_ROLE value: any - name: VAULT_IGNORE_MISSING_SECRETS value: 'false' - name: VAULT_ENV_PASSTHROUGH - name: VAULT_JSON_LOG value: 'false' - name: VAULT_CLIENT_TIMEOUT value: 30s - name: VAULT_LOG_LEVEL value: info - name: VAULT_CACERT value: /vault/tls/ca.crt
@bulky-electrician-72362 I make some test to create fresh new ingestion and never edit it ( so in database there is one record ) even in this case there are try to run twice.((
fresh cases
@bulky-electrician-72362 any thoughts?
b
you have
Copy code
- name: MCE_CONSUMER_ENABLED
         value: 'true'
       - name: MAE_CONSUMER_ENABLED
         value: 'true'
on gms , if you have MCE and MAE running separatly, disable them on gms
MCE and MEA can run in GMS or separatly, but not both, this causing them to consume events multiple times.
@brash-helicopter-28341 ^
b
@bulky-electrician-72362 Thx! Will try and come back
First try helps to solve twice run but seems to be another bug. I didn't see a progress of running on frontend UI. all ingestion frozen in failed status but in the logs there is a success. trying to fix it.
@bulky-electrician-72362 Thanks for helping. it works! huge thank you.
43 Views
Open in Slack
PreviousNext
Powered by