Hi <@U01NN9CTQ01> I found a weird mysql issue agai...
# all-things-deployment
m
Hi @early-lamp-41924 I found a weird mysql issue again 😭
After I made this change, the datahub-gms failed.
Copy code
diff --git a/docker/quickstart/docker-compose-dev.quickstart.yml b/docker/quickstart/docker-compose-dev.quickstart.yml
index fc1659488..f3a85db5b 100644
--- a/docker/quickstart/docker-compose-dev.quickstart.yml
+++ b/docker/quickstart/docker-compose-dev.quickstart.yml
@@ -61,14 +61,12 @@ services:
     - ${HOME}/.datahub/plugins:/etc/datahub/plugins
   datahub-gms:
     container_name: datahub-gms
-    depends_on:
-    - mysql
     environment:
     - DATASET_ENABLE_SCSI=false
-    - EBEAN_DATASOURCE_USERNAME=datahub
-    - EBEAN_DATASOURCE_PASSWORD=datahub
-    - EBEAN_DATASOURCE_HOST=mysql:3306
-    - EBEAN_DATASOURCE_URL=jdbc:<mysql://mysql:3306/datahub?verifyServerCertificate=false&useSSL=true&useUnicode=yes&characterEncoding=UTF-8&enabledTLSProtocols=TLSv1.2>
+    - EBEAN_DATASOURCE_USERNAME=myusername
+    - EBEAN_DATASOURCE_PASSWORD=mypassword
+    - EBEAN_DATASOURCE_HOST=<http://schemaregistrydevdb.proxysql.pinadmin.com:3306|schemaregistrydevdb.proxysql.pinadmin.com:3306>
+    - EBEAN_DATASOURCE_URL=jdbc:<mysql://schemaregistrydevdb.proxysql.pinadmin.com:3306/datahub?verifyServerCertificate=false&useSSL=true&useUnicode=yes&characterEncoding=UTF-8&enabledTLSProtocols=TLSv1.2>
     - EBEAN_DATASOURCE_DRIVER=com.mysql.jdbc.Driver
     - KAFKA_BOOTSTRAP_SERVER=broker:29092
     - KAFKA_SCHEMAREGISTRY_URL=<http://schema-registry:8081>
@@ -131,30 +129,13 @@ services:
     - KAFKA_BOOTSTRAP_SERVER=broker:29092
     hostname: kafka-setup
     image: linkedin/datahub-kafka-setup:${DATAHUB_VERSION:-head}
-  mysql:
-    command: --character-set-server=utf8mb4 --collation-server=utf8mb4_bin
-    container_name: mysql
-    environment:
-    - MYSQL_DATABASE=datahub
-    - MYSQL_USER=datahub
-    - MYSQL_PASSWORD=datahub
-    - MYSQL_ROOT_PASSWORD=datahub
-    hostname: mysql
-    image: mysql:5.7
-    ports:
-    - 3307:3306
-    volumes:
-    - ../mysql/init.sql:/docker-entrypoint-initdb.d/init.sql
-    - mysqldata:/var/lib/mysql
   mysql-setup:
     container_name: mysql-setup
-    depends_on:
-    - mysql
     environment:
-    - MYSQL_HOST=mysql
+    - MYSQL_HOST=<http://schemaregistrydevdb.proxysql.pinadmin.com|schemaregistrydevdb.proxysql.pinadmin.com>
     - MYSQL_PORT=3306
-    - MYSQL_USERNAME=datahub
-    - MYSQL_PASSWORD=datahub
+    - MYSQL_USERNAME=myusername
+    - MYSQL_PASSWORD=mypassword
     - DATAHUB_DB_NAME=datahub
     hostname: mysql-setup
     image: acryldata/datahub-mysql-setup:head
It failed with this error log
Copy code
2022-04-01 01:39:02.333:INFO:oejshC.ROOT:main: Initializing Spring root WebApplicationContext
01:39:02.350 [main] INFO  o.s.web.context.ContextLoader:271 - Root WebApplicationContext: initialization started
01:39:03.826 [main] INFO  o.s.c.s.PostProcessorRegistrationDelegate$BeanPostProcessorChecker:376 - Bean 'org.springframework.kafka.annotation.KafkaBootstrapConfiguration' of type [org.springframework.kafka.annotation.KafkaBootstrapConfiguration$$EnhancerBySpringCGLIB$$e952aacd] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying)
01:39:04.008 [main] INFO  io.ebean.EbeanVersion:31 - ebean version: 11.33.3
01:39:04.015 [main] INFO  i.e.config.properties.LoadContext:83 - loaded properties from [application.yml]
Loading class `com.mysql.jdbc.Driver'. This is deprecated. The new driver class is `com.mysql.cj.jdbc.Driver'. The driver is automatically registered via the SPI and manual loading of the driver class is generally unnecessary.
01:39:04.052 [main] INFO  i.e.datasource.pool.ConnectionPool:294 - DataSourcePool [gmsEbeanServiceConfig] autoCommit[false] transIsolation[READ_COMMITTED] min[2] max[50]
01:39:04.236 [main] INFO  i.e.d.pool.PooledConnectionQueue:405 - Reseting DataSourcePool [gmsEbeanServiceConfig] min[2] max[50] free[0] busy[0] waiting[0] highWaterMark[0] waitCount[0] hitCount[0]
01:39:04.237 [main] INFO  i.e.d.pool.PooledConnectionQueue:411 - Busy Connections:

01:39:04.239 [main] WARN  o.s.w.c.s.XmlWebApplicationContext:591 - Exception encountered during context initialization - cancelling refresh attempt: org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'recommendationServiceFactory': Unsatisfied dependency expressed through field 'topPlatformsCandidateSource'; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'topPlatformsCandidateSourceFactory': Unsatisfied dependency expressed through field 'entityService'; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'ebeanAspectDao' defined in com.linkedin.gms.factory.entity.EbeanAspectDaoFactory: Bean instantiation via factory method failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [com.linkedin.metadata.entity.ebean.EbeanAspectDao]: Factory method 'createInstance' threw exception; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'ebeanServer' defined in com.linkedin.gms.factory.entity.EbeanServerFactory: Bean instantiation via factory method failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [io.ebean.EbeanServer]: Factory method 'createServer' threw exception; nested exception is java.lang.NullPointerException
01:39:04.244 [main] ERROR o.s.web.context.ContextLoader:313 - Context initialization failed
org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'recommendationServiceFactory': Unsatisfied dependency expressed through field 'topPlatformsCandidateSource'; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'topPlatformsCandidateSourceFactory': Unsatisfied dependency expressed through field 'entityService'; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'ebeanAspectDao' defined in com.linkedin.gms.factory.entity.EbeanAspectDaoFactory: Bean instantiation via factory method failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [com.linkedin.metadata.entity.ebean.EbeanAspectDao]: Factory method 'createInstance' threw exception; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'ebeanServer' defined in com.linkedin.gms.factory.entity.EbeanServerFactory: Bean instantiation via factory method failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [io.ebean.EbeanServer]: Factory method 'createServer' threw exception; nested exception is java.lang.NullPointerException
	at org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor$AutowiredFieldElement.resolveFieldValue(AutowiredAnnotationBeanPostProcessor.java:659)
	at org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor$AutowiredFieldElement.inject(AutowiredAnnotationBeanPostProcessor.java:639)
	at org.springframework.beans.factory.annotation.InjectionMetadata.inject(InjectionMetadata.java:119)
	at org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor.postProcessProperties(AutowiredAnnotationBeanPostProcessor.java:399)
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.populateBean(AbstractAutowireCapableBeanFactory.java:1431)
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:619)
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:542)
	at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:335)
	at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:234)
	at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:333)
	at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:208)
	at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:953)
	at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:918)
	at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:583)
	at org.springframework.web.context.ContextLoader.configureAndRefreshWebApplicationContext(ContextLoader.java:401)
	at org.springframework.web.context.ContextLoader.initWebApplicationContext(ContextLoader.java:292)
	at org.springframework.web.context.ContextLoaderListener.contextInitialized(ContextLoaderListener.java:103)
	at org.eclipse.jetty.server.handler.ContextHandler.callContextInitialized(ContextHandler.java:921)
	at org.eclipse.jetty.servlet.ServletContextHandler.callContextInitialized(ServletContextHandler.java:554)
	at org.eclipse.jetty.server.handler.ContextHandler.startContext(ContextHandler.java:888)
	at org.eclipse.jetty.servlet.ServletContextHandler.startContext(ServletContextHandler.java:357)
	at org.eclipse.jetty.webapp.WebAppContext.startWebapp(WebAppContext.java:1443)
	at org.eclipse.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1407)
	at org.eclipse.jetty.server.handler.ContextHandler.doStart(ContextHandler.java:821)
	at org.eclipse.jetty.servlet.ServletContextHandler.doStart(ServletContextHandler.java:276)
	at org.eclipse.jetty.webapp.WebAppContext.doStart(WebAppContext.java:524)
	at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:72)
	at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169)
	at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:117)
	at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:106)
...
I tried to access my mysql instance by mysql cli
mysql -h <http://schemaregistrydevdb.proxysql.pinadmin.com|schemaregistrydevdb.proxysql.pinadmin.com>  -u myusername -p
It worked I also wrote a java script
Copy code
import java.sql.*;
  
public class FirstExample {
   // it works
   static final String DB_URL = "jdbc:<mysql://schemaregistrydevdb.proxysql.pinadmin.com:3306/datahub?verifyServerCertificate=false&useSSL=true&useUnicode=yes&characterEncoding=UTF-8&enabledTLSProtocols=TLSv1.2>";
   static final String USER = "myusername";
   static final String PASS = "mypassword";
   
   // it works too
   // static final String DB_URL = "jdbc:<mysql://localhost:3307/datahub?verifyServerCertificate=false&useSSL=true&useUnicode=yes&characterEncoding=UTF-8>";
   // static final String USER = "datahub";
   // static final String PASS = "datahub";
   static final String QUERY = "select * from datahub.metadata_aspect_v2;";

   public static void main(String[] args) {
      // Open a connection
      try(Connection conn = DriverManager.getConnection(DB_URL, USER, PASS);
         Statement stmt = conn.createStatement();
         ResultSet rs = stmt.executeQuery(QUERY);) {
         // Extract data from result set
         while (rs.next()) {
            // Retrieve by column name
            System.out.print("urn: " + rs.getString("urn"));
         }
      } catch (SQLException e) {
         e.printStackTrace();
      }
   }
}
Do you know what does the NPE in Ebean mean?
e
NPE on ebean server means there is something wrong with the connection to the local db
did you run mysql-setup-job in your local db?
Haha feel like a dejavu
m
Yes
e
but need to check
and username and password is correct for both containers?
m
Yes
e
But this worked before right? what changed?
m
I ran
Copy code
curl --location --request DELETE '<http://my_remote_schema_registry:8081/subjects/schema_registry_metadata_audit_event-value>'
But this should not be related. zookeeper and schema-registry are local services
FYI, I found the connection between EBean and our mysql instance is very flaky. Sometimes it does not work, but if I refresh page again, it may work again. Now it is totally bad. I want to prove it is our mysql instance issue. But mysql cli and my java script are good.
I also repeated my script 100 times, all passed.
This is a log when connection is flaky https://gist.githubusercontent.com/wizardxz/25308db6461d02f00e66ff618c302ad3/raw/fd97c109bb1e2c1e0329a3cc13dc6e5179e51817/gistfile1.txt
Copy code
javax.persistence.PersistenceException: java.sql.SQLNonTransientConnectionException: Could not connect to address=(host=<http://schemaregistrydevdb.proxysql.pinadmin.com|schemaregistrydevdb.proxysql.pinadmin.com>)(port=3306)(type=master) : Could not connect to <http://schemaregistrydevdb.proxysql.pinadmin.com:3306|schemaregistrydevdb.proxysql.pinadmin.com:3306> : unexpected end of stream, read 0 bytes from 4 (socket was closed by server)
Our mysql team found a suspicious. Will update the progress here. Thank you
Just FYI, our mysql team confirmed it may be a bug related to 8.0 or mtls. Currently they applied a short term fix on it. and it works! Thanks!
e
aewsome!!
glad it worked out!