We ingest quite a lot of looker dashboard elements, and started to see some issues regarding that in the GMS logs:

07:10:21.041 [qtp544724190-10] INFO  c.l.metadata.entity.EntityService:681 - INGEST urn urn:li:chart:(looker,dashboard_elements.21845) with system metadata {lastObserved=1645427421035, runId=looker-2022_02_21-07_10_08}
07:10:21.053 [qtp544724190-10] INFO  c.l.m.filter.RestliLoggingFilter:56 - POST /entities?action=ingest - ingest - 500 - 12ms
07:10:21.053 [qtp544724190-10] ERROR c.l.m.filter.RestliLoggingFilter:38 - com.datahub.util.exception.RetryLimitReached: Failed to add after 3 retries

Is this a problem with the GMS database?

@high-hospital-85984 We have found that reducing

max_threads

to a low number (e.g: 3) fixes this issue. There might be some new rate limits introduced on Looker APIs. cc @dazzling-judge-80093

You mean in the ingestion recipe? I can try that, but I'm more concerned about the collaboration between the MCE and GMS services. There is no hurry for the MCE to ingest updates from Kafka, but still it manages to overload the GMS (I'm just guessing that's what's happening).

sorry - didn't notice that the log snippet is from GMS

Are you referring to retry logic? That was only whem using datahub-rest not datahub-kafka

let's track this issue closely cc @big-carpet-38439

I can do some more digging on my end as well. I just stumbled on this while investigating another issue, so didn't dive too deep yet.

This is coming in the Entity Service when we attempt to write into MySQL. Something is failing potentially due to the DB connections being oversaturated. Are you using hosted MySQL?

Hosted Postgres on AWS,

db.t3.medium

. I need to add better monitoring on the whole deployment to be give you better answers