Hello team, I have a datahub setup on K8s cluster running on AWS, using some AWS managed services like RDS, MSK, Opensearch, etc… It run fine at first but after a while (couple of day) it stop working, when I see log, the gms pod is failing with this weird log>

kubectl get pods -n my-datahub                                          
NAME                                             READY   STATUS             RESTARTS          AGE
my-acryl-datahub-actions-7f7dbcb7cb-jwlm7   0/1     CrashLoopBackOff   126 (2m39s ago)   18h
my-cp-schema-registry-5cbf4478f-2xgnt       2/2     Running            0                 51m
my-datahub-frontend-795fb7dd7d-qj7p9        1/1     Running            0                 52m
my-datahub-gms-7466d54b7-5hwxz              0/1     CrashLoopBackOff   194 (4m55s ago)   17h

kubectl logs my-datahub-gms-7466d54b7-5hwxz  -n my-datahub  
+ echo
+ grep -q ://
+ NEO4J_HOST=http://
+ [[ ! -z datahubes ]]
+ [[ -z '' ]]
++ base64 --wrap 0
++ echo -ne 'username:password'
+ AUTH_TOKEN=username:password
+ ELASTICSEARCH_AUTH_HEADER='Authorization:Basic username:password'
+ [[ -z Authorization:Basic username:password ]]
+ [[ true == true ]]
+ ELASTICSEARCH_PROTOCOL=https
+ WAIT_FOR_EBEAN=
+ [[ '' != true ]]
+ [[ '' == ebean ]]
+ [[ -z '' ]]
+ WAIT_FOR_EBEAN=' -wait <tcp://my-datahub-dev-ue1.cluster-xxxxxxxxx.us-east-1.rds.amazonaws.com:3306> '
+ WAIT_FOR_CASSANDRA=
+ [[ '' == cassandra ]]
+ WAIT_FOR_KAFKA=
+ [[ '' != true ]]
++ echo <http://b-1.mydatahub.xxxxx.c5.kafka.us-east-1.amazonaws.com:9096,b-2.mydatahub.xxxxx.c5.kafka.us-east-1.amazonaws.com:9096,b-3.mydatahub.xxxxx.c5.kafka.us-east-1.amazonaws.com:9096|b-1.mydatahub.xxxxx.c5.kafka.us-east-1.amazonaws.com:9096,b-2.mydatahub.xxxxx.c5.kafka.us-east-1.amazonaws.com:9096,b-3.mydatahub.xxxxx.c5.kafka.us-east-1.amazonaws.com:9096>
++ sed 's/,/ -wait tcp:\/\//g'
+ WAIT_FOR_KAFKA=' -wait <tcp://b-1.mydatahub.xxxxx.c5.kafka.us-east-1.amazonaws.com:9096> -wait <tcp://b-2.mydatahub.xxxxx.c5.kafka.us-east-1.amazonaws.com:9096> -wait <tcp://b-3.mydatahub.xxxxx.c5.kafka.us-east-1.amazonaws.com:9096> '
+ WAIT_FOR_NEO4J=
+ [[ elasticsearch != elasticsearch ]]
+ OTEL_AGENT=
+ [[ '' == true ]]
+ PROMETHEUS_AGENT=
+ [[ true == true ]]
+ PROMETHEUS_AGENT='-javaagent:jmx_prometheus_javaagent.jar=4318:/datahub/datahub-gms/scripts/prometheus-config.yaml '
+ auth_resource_dir=/etc/datahub/plugins/auth/resources
+ CLASSES_DIR=
+ [[ '' == true ]]
+ COMMON='
     -wait <tcp://my-datahub-dev-ue1.cluster-xxxxxx.us-east-1.rds.amazonaws.com:3306>            -wait <tcp://b-1.mydatahub.xxxxx.c5.kafka.us-east-1.amazonaws.com:9096> -wait <tcp://b-2.mydatahub.xxxxx.c5.kafka.us-east-1.amazonaws.com:9096> -wait <tcp://b-3.mydatahub.xxxxx.c5.kafka.us-east-1.amazonaws.com:9096>           -timeout 240s     java            -javaagent:jmx_prometheus_javaagent.jar=4318:/datahub/datahub-gms/scripts/prometheus-config.yaml      -jar /jetty-runner.jar     --jar jetty-util.jar     --jar jetty-jmx.jar      --config /datahub/datahub-gms/scripts/jetty.xml     /datahub/datahub-gms/bin/war.war'
+ [[ false != true ]]
+ exec dockerize -wait <https://vpc-my-datahub-xxxxxx.us-east-1.es.amazonaws.com:443> -wait-http-header 'Authorization:Basic ZGF0YWh1YmVzOlhjTyEyNlNPKmI6VThQOmxyLTJnTjZOZDYwNXQ3PU0rK2l7PA==' -wait <tcp://my-datahub-dev-ue1.cluster-xxxxxx.us-east-1.rds.amazonaws.com:3306> -wait <tcp://b-1.mydatahub.xxxxx.c5.kafka.us-east-1.amazonaws.com:9096> -wait <tcp://b-2.mydatahub.xxxxx.c5.kafka.us-east-1.amazonaws.com:9096> -wait <tcp://b-3.mydatahub.xxxxx.c5.kafka.us-east-1.amazonaws.com:9096> -timeout 240s java -javaagent:jmx_prometheus_javaagent.jar=4318:/datahub/datahub-gms/scripts/prometheus-config.yaml -jar /jetty-runner.jar --jar jetty-util.jar --jar jetty-jmx.jar --config /datahub/datahub-gms/scripts/jetty.xml /datahub/datahub-gms/bin/war.war
2023/02/09 18:31:19 Waiting for: <https://vpc-my-datahub-xxxxxx.us-east-1.es.amazonaws.com:443>
2023/02/09 18:31:19 Waiting for: <tcp://my-datahub-dev-ue1.cluster-xxxxxx.us-east-1.rds.amazonaws.com:3306>
2023/02/09 18:31:19 Waiting for: <tcp://b-1.mydatahub.xxxxx.c5.kafka.us-east-1.amazonaws.com:9096>
2023/02/09 18:31:19 Waiting for: <tcp://b-2.mydatahub.xxxxx.c5.kafka.us-east-1.amazonaws.com:9096>
2023/02/09 18:31:19 Waiting for: <tcp://b-3.mydatahub.xxxxx.c5.kafka.us-east-1.amazonaws.com:9096>
2023/02/09 18:31:19 Problem with dial: dial tcp 10.194.49.47:9096: connect: connection refused. Sleeping 1s
2023/02/09 18:31:19 Connected to <tcp://b-2.mydatahub.xxxxx.c5.kafka.us-east-1.amazonaws.com:9096>
2023/02/09 18:31:19 Connected to <tcp://my-datahub-dev-ue1.cluster-xxxxxx.us-east-1.rds.amazonaws.com:3306>
2023/02/09 18:31:19 Connected to <tcp://b-3.mydatahub.xxxxx.c5.kafka.us-east-1.amazonaws.com:9096>
2023/02/09 18:31:20 Received 200 from <https://vpc-my-datahub-xxxxxx.us-east-1.es.amazonaws.com:443>
2023/02/09 18:31:20 Problem with dial: dial tcp 10.194.49.47:9096: connect: connection refused. Sleeping 1s
2023/02/09 18:31:21 Problem with dial: dial tcp 10.194.49.47:9096: connect: connection refused. Sleeping 1s
....... lots of log look like this .................
2023/02/09 18:33:27 Problem with dial: dial tcp 10.194.49.47:9096: connect: connection refused. Sleeping 1s
2023/02/09 18:33:28 Problem with dial: dial tcp 10.194.49.47:9096: connect: connection refused. Sleeping 1s
2023/02/09 18:33:29 Problem with dial: dial tcp 10.194.49.47:9096: connect: connection refused. Sleeping 1s
2023/02/09 18:33:30 Problem with dial: dial tcp 10.194.49.47:9096: connect: connection refused. Sleeping 1s
2023/02/09 18:33:31 Problem with dial: dial tcp 10.194.49.47:9096: connect: connection refused. Sleeping 1s
2023/02/09 18:33:32 Problem with dial: dial tcp 10.194.49.47:9096: connect: connection refused. Sleeping 1s
2023/02/09 18:35:18 Problem with dial: dial tcp 10.194.49.47:9096: connect: connection refused. Sleeping 1s
2023/02/09 18:35:19 Problem with dial: dial tcp 10.194.49.47:9096: connect: connection refused. Sleeping 1s
2023/02/09 18:35:19 Timeout after 4m0s waiting on dependencies to become available: [<https://vpc-my-datahub-xxxxxx.us-east-1.es.amazonaws.com:443> <tcp://my-datahub-dev-ue1.cluster-xxxxxx.us-east-1.rds.amazonaws.com:3306> <tcp://b-1.mydatahub.xxxxx.c5.kafka.us-east-1.amazonaws.com:9096> <tcp://b-2.mydatahub.xxxxx.c5.kafka.us-east-1.amazonaws.com:9096> <tcp://b-3.mydatahub.xxxxx.c5.kafka.us-east-1.amazonaws.com:9096>]

so what is wrong with it?

Hi @wooden-hamburger-59537, what version of DataHub are you currently running?

it’s

v0.9.5

but look like this related to our security group setup. weird that it work at first and started to fail lateer