👋 Hello, team! our GMS instances occasionally hit errors trying to create an Elasticsearch system metadata index. Error looks like:

Failed to instantiate [com.linkedin.metadata.kafka.hook.UpdateIndicesHook]: Constructor threw exception; nested exception is java.lang.RuntimeException: Could not configure system metadata index
22:15:58 [main] INFO  c.l.r.t.h.c.c.AbstractNettyClient - Shutdown requested
22:15:58 [main] INFO  c.l.r.t.h.c.c.AbstractNettyClient - Shutting down
...
Caused by: java.net.ConnectException: Connection refused

any idea on what might cause it and how to fix it👀? thanks! cc @bitter-lizard-32293 @numerous-byte-87938

Seems like we hit this every time we deploy GMS. When we restart the instance things are fine. So not sure if it's just something that needs more retries or staggering our pods during deployment