Hey folks! I have a Looker ingestion job that prev...
# all-things-deploymentb
brief-dog-31091
12/07/2023, 10:01 PMHey folks! I have a Looker ingestion job that previously ran, but now I'm getting the following error:
Copy code
[{'error': 'Unable to emit metadata to DataHub GMS',
'info': {'message': "HTTPConnectionPool(host='datahub-datahub-gms', port=8080): Max retries exceeded with url: "
"/entities?action=ingest (Caused by ResponseError('too many 500 error responses'))",
'id': 'urn:li:dataset:(urn:li:dataPlatform:looker,shiphero.explore.fulfillment_members,PROD)'}}]o
orange-night-91387
12/07/2023, 10:42 PMIt looks like it's hitting 500 errors on the GMS side, have you inspected the GMS side logs to see what the source of the 500 error is?
b
brief-dog-31091
12/19/2023, 5:08 PMHi @orange-night-91387! Sorry for the delay here. Got distracted with some other data engineering objectives...
Here's what I'm seeing in the logs on the GMS side:
o
orange-night-91387
12/19/2023, 5:11 PMHmm, there isn't a stacktrace in the area of this log?
b
brief-dog-31091
12/19/2023, 5:14 PMDoesn't seem to be, sadly.
brief-dog-31091
12/19/2023, 5:16 PMI may need to log into the deployment and see if I can get more detail that way.
👍 1
brief-dog-31091
12/19/2023, 5:22 PMAny hypotheses as to what could be happening here? May take me a bit to dig in here.
o
orange-night-91387
12/19/2023, 5:48 PMUnfortunately no, basically all this tells me is that something went wrong with the ingestion and the server threw an error when trying to process it... which you already know 😅 500 is a generic server error so there should be some stacktrace thrown during execution of the ingestion that would give more information
thank you 1
orange-night-91387
12/19/2023, 5:49 PMCould be anything from the server isn't correctly configured, is experiencing problems from the storage layer, the message is malformatted, etc
b
brief-dog-31091
12/19/2023, 9:24 PMGood to know. I'll update this thread once I find valuable troubleshooting data
brief-dog-31091
12/19/2023, 9:31 PMAh, ha! Found it, @orange-night-91387. Seems like it's an issue with MetadataChangeLog, I think?
o
orange-night-91387
12/19/2023, 9:50 PMInteresting, seems like an intermittent problem with your Kafka connection. The producer is failing to produce messages to the MCL topic
orange-night-91387
12/19/2023, 9:52 PMYour ingestion speeds are really slow which makes me think that there is a long network delay between GMS and your Kafka instance, are GMS and Kafka in the same region? Or is this a cross region cluster?
orange-night-91387
12/19/2023, 9:53 PM Copy code
2023-12-19 17:03:19,400 [pool-13-thread-2] INFO c.l.m.filter.RestliLoggingFilter:55 - POST /entities?action=ingest - ingest - 200 - 5984msorange-night-91387
12/19/2023, 9:54 PMand the one that failed failed after 13s, which might be hitting the backoff/retry limit or something
b
brief-dog-31091
12/19/2023, 10:04 PMAll in the same region, us-east4.
brief-dog-31091
12/19/2023, 10:06 PMThat is, both GMS and Kafka are in the same region.
o
orange-night-91387
12/19/2023, 10:12 PMWeird, are they in different VPCs or having to cross any other network boundary?
orange-night-91387
12/19/2023, 10:14 PMIf it’s not explainable by network definitely check the broker’s metrics and see if it’s getting overloaded
b
brief-dog-31091
12/19/2023, 10:25 PMOur deployment is in GCP.
What's really weird is that when I initially set up the Looker ingestion, it worked just fine and then all of the sudden it just started failing. Not great intel for troubleshooting, but again, it's worth noting that when it was setup initially everything was fine.
How do I check the broker metrics?
o
orange-night-91387
12/19/2023, 10:36 PMWith the prerequisites broker you’ll only have the default metrics available, but you can at minimum check how the pod is looking from a cpu and memory perspective. Also worth looking at the logs to see if anything looks off.
It’s also worth noting that generally the prerequisites stuff is there to get a test instance up and for production load it’s recommended to use a managed service for Kafka unless your team is just really experienced with managing the external services in production.
👍 1
b
brief-dog-31091
12/26/2023, 4:55 PMHope you enjoyed your holiday, @orange-night-91387!
I haven't found anything abnormal in the logs, and the pods looks okay from a cpu and memory perspective. It is curious how the pipeline works until it gets to this one Explore:
urn:li:dataPlatform:looker,shiphero.explore.fulfillment_members,PRODbrief-dog-31091
01/02/2024, 9:35 PMSeparate from the ingestion issue, without making any changes to the cluster, I'm now having issues using Google Sign-in:
Copy code
Retryable PersistenceException: Error when batch flush on sql: insert into metadata_aspect_v2 (urn, aspect, version, metadata, createdOn, createdBy, createdFor, systemmetadata) values (?,?,?,?,?,?,?,?)brief-dog-31091
01/02/2024, 9:35 PMI tried restarting the deployment but to no avail.