Hi Team, I have deployed on kubernetes using helm chart for the last version 10.2 with the default configuration. MySQL, however, is constantly restarting. Last 13 days is restarted 44 times. Does anyone have any idea why is this happening?

NAME                                                READY   STATUS      RESTARTS        AGE
datahub-acryl-datahub-actions-867f6bb5d4-kpq57      1/1     Running     0               13d
datahub-cli-65548b99c7-bx7v9                        1/1     Running     0               30h
datahub-datahub-frontend-7c5bf5654b-xbhmq           1/1     Running     0               13d
datahub-datahub-gms-56f8848747-t2gwp                1/1     Running     1 (4h42m ago)   13d
datahub-datahub-system-update-job-clcgs             0/1     Completed   0               13d
datahub-kafka-setup-job-b5rxq                       0/1     Completed   0               13d
datahub-mysql-setup-job-cg4gb                       0/1     Completed   0               13d
datahub-nocode-migration-job-rgsbn                  0/1     Completed   0               13d
elasticsearch-master-0                              1/1     Running     4 (174m ago)    30h
prerequisites-cp-schema-registry-6f4b5b894f-kvh5p   2/2     Running     0               13d
prerequisites-kafka-0                               1/1     Running     0               30h
prerequisites-mysql-0                               1/1     Running     43 (11m ago)    30h
prerequisites-zookeeper-0                           1/1     Running     0               30h
prerequisites-mysql-0                               0/1     Running     44 (3s ago)     30h

describing the pod:

Type     Reason     Age                 From     Message
  ----     ------     ----                ----     -------
  Warning  Unhealthy  47m (x39 over 8h)   kubelet  Startup probe failed: command "/bin/bash -ec password_aux=\"${MYSQL_ROOT_PASSWORD:-}\"\nif [[ -f \"${MYSQL_ROOT_PASSWORD_FILE:-}\" ]]; then\n    password_aux=$(cat \"$MYSQL_ROOT_PASSWORD_FILE\")\nfi\nmysqladmin status -uroot -p\"${password_aux}\"\n" timed out
  Warning  Unhealthy  32m (x123 over 9h)  kubelet  Startup probe failed: mysqladmin: [Warning] Using a password on the command line interface can be insecure.
mysqladmin: connect to server at 'localhost' failed
error: 'Can't connect to local MySQL server through socket '/opt/bitnami/mysql/tmp/mysql.sock' (2)'
Check that mysqld is running and that the socket: '/opt/bitnami/mysql/tmp/mysql.sock' exists!
  Warning  Unhealthy  6m43s (x436 over 30h)  kubelet  Readiness probe failed: command "/bin/bash -ec password_aux=\"${MYSQL_ROOT_PASSWORD:-}\"\nif [[ -f \"${MYSQL_ROOT_PASSWORD_FILE:-}\" ]]; then\n    password_aux=$(cat \"$MYSQL_ROOT_PASSWORD_FILE\")\nfi\nmysqladmin status -uroot -p\"${password_aux}\"\n" timed out
  Warning  Unhealthy  2m43s (x428 over 30h)  kubelet  Liveness probe failed: command "/bin/bash -ec password_aux=\"${MYSQL_ROOT_PASSWORD:-}\"\nif [[ -f \"${MYSQL_ROOT_PASSWORD_FILE:-}\" ]]; then\n    password_aux=$(cat \"$MYSQL_ROOT_PASSWORD_FILE\")\nfi\nmysqladmin status -uroot -p\"${password_aux}\"\n" timed out

There is no meaningful information in the logs:

2023-05-26T22:21:42.762925Z 0 [System] [MY-010931] [Server] /opt/bitnami/mysql/bin/mysqld: ready for connections. Version: '8.0.29'  socket: '/opt/bitnami/mysql/tmp/mysql.sock'  port: 3306  Source distribution.
2023-05-26T22:38:42.603109Z 0 [System] [MY-013172] [Server] Received SHUTDOWN from user <via user signal>. Shutting down mysqld (Version: 8.0.29).
2023-05-26T22:38:44.996744Z 0 [Warning] [MY-010909] [Server] /opt/bitnami/mysql/bin/mysqld: Forcing close of thread 56  user: 'root'.
2023-05-26T22:38:44.996858Z 0 [Warning] [MY-010909] [Server] /opt/bitnami/mysql/bin/mysqld: Forcing close of thread 57  user: 'root'.
2023-05-26T22:38:44.996887Z 0 [Warning] [MY-010909] [Server] /opt/bitnami/mysql/bin/mysqld: Forcing close of thread 58  user: 'root'.
2023-05-26T22:38:44.996913Z 0 [Warning] [MY-010909] [Server] /opt/bitnami/mysql/bin/mysqld: Forcing close of thread 59  user: 'root'.
2023-05-26T22:38:44.996936Z 0 [Warning] [MY-010909] [Server] /opt/bitnami/mysql/bin/mysqld: Forcing close of thread 124  user: 'root'.
2023-05-26T22:38:44.996962Z 0 [Warning] [MY-010909] [Server] /opt/bitnami/mysql/bin/mysqld: Forcing close of thread 60  user: 'root'.
2023-05-26T22:38:44.996999Z 0 [Warning] [MY-010909] [Server] /opt/bitnami/mysql/bin/mysqld: Forcing close of thread 61  user: 'root'.
2023-05-26T22:38:44.997025Z 0 [Warning] [MY-010909] [Server] /opt/bitnami/mysql/bin/mysqld: Forcing close of thread 62  user: 'root'.
2023-05-26T22:38:44.997048Z 0 [Warning] [MY-010909] [Server] /opt/bitnami/mysql/bin/mysqld: Forcing close of thread 54  user: 'root'.
2023-05-26T22:38:44.997079Z 0 [Warning] [MY-010909] [Server] /opt/bitnami/mysql/bin/mysqld: Forcing close of thread 23  user: 'root'.
2023-05-26T22:38:44.997101Z 0 [Warning] [MY-010909] [Server] /opt/bitnami/mysql/bin/mysqld: Forcing close of thread 55  user: 'root'.
2023-05-26T22:38:48.845805Z 0 [System] [MY-010910] [Server] /opt/bitnami/mysql/bin/mysqld: Shutdown complete (mysqld 8.0.29)  Source distribution.

@orange-night-91387 might help you

Whenever you deployed, did you do a full redeploy including the prerequisites? It seems the MySQL docker container can get into this state if the PVCs do not align with the deployed container: https://github.com/bitnami/charts/issues/9083

Is there a full solution to this or is recreate the pvc the solution everytime?

Hey there! 👋 Make sure your message includes the following information if relevant, so we can help more effectively! 1. Which DataHub version are you using? (e.g. 0.12.0) 2. Please post any relevant error logs on the thread!