Heyo, From nowhere my gms pod started failing and logs are showing:

Caused by: com.linkedin.r2.RetriableRequestException: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: localhost/127.0.0.1:8080
│ 2023-04-02 10:21:19,146 [pool-14-thread-1] ERROR c.d.m.ingestion.IngestionScheduler:244 - Failed to retrieve ingestion sources! Skipping updating schedule cache until next refresh. start: 0, count: 30                                                                     ││ com.linkedin.r2.RemoteInvocationException: com.linkedin.r2.RemoteInvocationException: Failed to get response from server for URI <http://localhost:8080/entities>
2023-04-02 10:21:19,143 [R2 Nio Event Loop-1-1] WARN  c.l.r.t.h.c.c.ChannelPoolLifecycle:139 - Failed to create channel, remote=localhost/127.0.0.1:8080

My upgrade job [1] (deployed with helm chart actual version is 0.2.160 / 10.0.0 (app version) My first thought was to upgrade the helm version to 0.2.161 but does anyone know the problem’s source? On the instance, there were several ingestions nothing big and I was playing with graphql api + custom python script for groups and policies I found similar cases:

<https://datahubspace.slack.com/archives/C029A3M079U/p1678354501041149>
<https://datahubspace.slack.com/archives/CV2UVAPPG/p1678293436288999>

Hey there 👋 I'm The DataHub Community Support bot. I'm here to help make sure the community can best support you with your request. Let's double check a few things first: ✅ There's a lot of good information on our docs site: www.datahubproject.io/docs, Have you searched there for a solution? ✅ It's not uncommon that someone has run into your exact problem before in the community. Have you searched Slack for similar issues? Did you find a solution to your issue? ❌ Sorry you weren't able to find a solution. I'm sending you some tips on info you can provide to help the community troubleshoot. Whenever you feel your issue is solved, please react ✅ to your original message to let us know!

1. 0.2.160 2. helm chart 3. in the thread

Have you run the datahub-upgrade job? https://datahubproject.io/docs/docker/development/#system-update

Updating to the latest version 0.2.161 (10.1) made the problem go away but my question Is why this happened Deployment stayed the same but only the version of helm chart has been bumped and from nowhere gms crashed with the error from my previous thread, DataHub is being deployed on k8s, do you have any suggestion on how to prevent these actions or how to properly fix when this happened once again? I assume that the upgrading job here is doing all the good, is there any change to synthetically induce this job? cc @astonishing-answer-96712

My guess is it may have had something to do with the restore indices job running in upgrade

@astonishing-answer-96712 I will try to revive this thread because from nowhere on

0.2.161

version gms started failing with the same errors like above

2023-05-18 08:01:32,548 [pool-15-thread-1] ERROR c.d.m.ingestion.IngestionScheduler:244 - Failed to retrieve ingestion sources! Ski ││ pping updating schedule cache until next refresh. start: 0, count: 30                                                               │
│ com.linkedin.r2.RemoteInvocationException: com.linkedin.r2.RemoteInvocationException: Failed to get response from server for URI ht ││ <tp://localhost:8080/entities>

2023-05-18 08:05:32,942 [I/O dispatcher 1] ERROR c.l.m.s.e.update.BulkListener:56 - Error feeding bulk request. No retries left. Re │
│ java.io.IOException: Unable to parse response body for Response{requestLine=POST /_bulk?timeout=1m HTTP/1.1, host=