@here Today, we are seeing these errors in GMS ser...
# all-things-deployment
g
@here Today, we are seeing these errors in GMS service logs when frontend tries to connect with GMS service. It was all working yesterday. But today we are seeing these errors. We are using aurora backend. And gms service is running on spot instances. Can anyone help?
Copy code
04:52:07.924 [Thread-92] ERROR c.l.d.g.e.DataHubDataFetcherExceptionHandler:21 - Failed to execute DataFetcher
java.util.concurrent.CompletionException: javax.persistence.PersistenceException: java.sql.SQLNonTransientConnectionException: Could not connect to address=(host=<redacted>)(port=3306)(type=master) : Could not connect to <redacted>:3306 : PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
	at java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:273)
	at java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:280)
	at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1606)
	at java.lang.Thread.run(Thread.java:748)
Caused by: javax.persistence.PersistenceException: java.sql.SQLNonTransientConnectionException: Could not connect to address=(host=<redacted>)(port=3306)(type=master) : Could not connect to <redacted>:3306 : PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
	at io.ebeaninternal.server.transaction.TransactionFactoryBasic.createQueryTransaction(TransactionFactoryBasic.java:35)
	at io.ebeaninternal.server.transaction.TransactionManager.createQueryTransaction(TransactionManager.java:360)
	at io.ebeaninternal.server.core.DefaultServer.createQueryTransaction(DefaultServer.java:2306)
	at io.ebeaninternal.server.core.OrmQueryRequest.initTransIfRequired(OrmQueryRequest.java:282)
	at io.ebeaninternal.server.core.DefaultServer.findList(DefaultServer.java:1595)
	at io.ebeaninternal.server.core.DefaultServer.findList(DefaultServer.java:1574)
	at io.ebeaninternal.server.querydefn.DefaultOrmQuery.findList(DefaultOrmQuery.java:1481)
	at com.linkedin.metadata.entity.ebean.EbeanAspectDao.batchGetUnion(EbeanAspectDao.java:359)
	at com.linkedin.metadata.entity.ebean.EbeanAspectDao.batchGet(EbeanAspectDao.java:279)
	at com.linkedin.metadata.entity.ebean.EbeanAspectDao.batchGet(EbeanAspectDao.java:260)
	at com.linkedin.metadata.entity.EntityService.getEnvelopedAspects(EntityService.java:1504)
	at com.linkedin.metadata.entity.EntityService.getCorrespondingAspects(EntityService.java:353)
	at com.linkedin.metadata.entity.EntityService.getLatestEnvelopedAspects(EntityService.java:307)
	at com.linkedin.metadata.entity.EntityService.getEntitiesV2(EntityService.java:263)
	at com.linkedin.entity.client.JavaEntityClient.batchGetV2(JavaEntityClient.java:103)
	at com.linkedin.datahub.graphql.resolvers.MeResolver.lambda$get$0(MeResolver.java:52)
	at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1604)
	... 1 common frames omitted
It appeared first time when it tried to reset the connection. Posting earlier exceptions
Copy code
10:10:19.060 [qtp1830908236-14] INFO  c.l.m.r.entity.EntityResource:310 - GET LIST RESULTS for dataHubIngestionSource with filter {or=[{and=[]}]}
10:10:19.103 [qtp1830908236-14] INFO  c.l.parseq.TaskDescriptorFactory:44 - No provider found for TaskDescriptor, falling back to DefaultTaskDescriptor
10:10:19.158 [pool-11-thread-1] INFO  c.l.m.filter.RestliLoggingFilter:55 - POST /entities?action=list - list - 200 - 103ms
10:10:19.213 [pool-11-thread-1] INFO  c.l.m.filter.RestliLoggingFilter:55 - GET /entitiesV2?aspects=List(dataHubIngestionSourceInfo)&ids=List() - batchGet - 200 - 9ms
10:10:45.505 [Thread-15] INFO  c.l.m.b.s.RestoreDbtSiblingsIndices:75 - RestoreDbtSiblingsIndices has run before with this version. Skipping
10:12:25.666 [pool-11-thread-1] INFO  c.l.m.filter.RestliLoggingFilter:55 - GET /entitiesV2?ids=List(urn%3Ali%3Acorpuser%3Adatahub) - batchGet - 200 - 15ms
10:12:25.715 [pool-11-thread-1] INFO  c.l.m.filter.RestliLoggingFilter:55 - GET /entitiesV2?ids=List(urn%3Ali%3Acorpuser%3Adatahub) - batchGet - 200 - 5ms
22:10:18.397 [qtp1830908236-11] INFO  c.l.m.r.entity.EntityResource:310 - GET LIST RESULTS for dataHubIngestionSource with filter {or=[{and=[]}]}
22:10:18.407 [pool-11-thread-1] INFO  c.l.m.filter.RestliLoggingFilter:55 - POST /entities?action=list - list - 200 - 11ms
22:10:18.418 [pool-11-thread-1] INFO  c.l.m.filter.RestliLoggingFilter:55 - GET /entitiesV2?aspects=List(dataHubIngestionSourceInfo)&ids=List() - batchGet - 200 - 6ms
03:11:09.692 [gmsEbeanServiceConfig.heartBeat] INFO  i.e.d.pool.PooledConnectionQueue:405 - Reseting DataSourcePool [gmsEbeanServiceConfig] min[2] max[50] free[1] busy[0] waiting[0] highWaterMark[3] waitCount[0] hitCount[2696]
03:11:09.694 [gmsEbeanServiceConfig.heartBeat] INFO  i.e.d.pool.PooledConnectionQueue:411 - Busy Connections:

03:11:09.698 [gmsEbeanServiceConfig.heartBeat] ERROR i.e.datasource.pool.ConnectionPool:444 - Error trying to trim idle connections - message:null
java.lang.NullPointerException: null
	at io.ebean.datasource.pool.ConnectionPool.notifyDataSourceIsDown(ConnectionPool.java:406)
	at io.ebean.datasource.pool.ConnectionPool.createUnpooledConnection(ConnectionPool.java:535)
	at io.ebean.datasource.pool.ConnectionPool.createUnpooledConnection(ConnectionPool.java:524)
	at io.ebean.datasource.pool.ConnectionPool.createConnectionForQueue(ConnectionPool.java:766)
	at io.ebean.datasource.pool.PooledConnectionQueue.ensureMinimumConnections(PooledConnectionQueue.java:227)
	at io.ebean.datasource.pool.PooledConnectionQueue.trim(PooledConnectionQueue.java:424)
	at io.ebean.datasource.pool.ConnectionPool.trimIdleConnections(ConnectionPool.java:441)
	at io.ebean.datasource.pool.ConnectionPool.checkDataSource(ConnectionPool.java:459)
	at io.ebean.datasource.pool.ConnectionPool.access$000(ConnectionPool.java:43)
	at io.ebean.datasource.pool.ConnectionPool$HeartBeatRunnable.run(ConnectionPool.java:260)
	at java.util.TimerThread.mainLoop(Timer.java:555)
	at java.util.TimerThread.run(Timer.java:505)
03:12:50.348 [pool-5-thread-1] ERROR c.d.authorization.DataHubAuthorizer:229 - Failed to retrieve policy urns! Skipping updating policy cache until next refresh. start: 0, count: 30
javax.persistence.PersistenceException: java.sql.SQLNonTransientConnectionException: Could not connect to address=(host=<redacted>)(port=3306)(type=master) : Could not connect to <redacted>:3306 : PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
	at io.ebeaninternal.server.transaction.TransactionFactoryBasic.createQueryTransaction(TransactionFactoryBasic.java:35)
	at io.ebeaninternal.server.transaction.TransactionManager.createQueryTransaction(TransactionManager.java:360)
	at io.ebeaninternal.server.core.DefaultServer.createQueryTransaction(DefaultServer.java:2306)
	at io.ebeaninternal.server.core.OrmQueryRequest.initTransIfRequired(OrmQueryRequest.java:282)
	at io.ebeaninternal.server.core.DefaultServer.findList(DefaultServer.java:1595)
	at io.ebeaninternal.server.core.DefaultServer.findList(DefaultServer.java:1574)
	at io.ebeaninternal.server.querydefn.DefaultOrmQuery.findList(DefaultOrmQuery.java:1481)
	at com.linkedin.metadata.entity.ebean.EbeanAspectDao.batchGetUnion(EbeanAspectDao.java:359)
	at com.linkedin.metadata.entity.ebean.EbeanAspectDao.batchGet(EbeanAspectDao.java:279)
	at com.linkedin.metadata.entity.ebean.EbeanAspectDao.batchGet(EbeanAspectDao.java:260)
	at com.linkedin.metadata.entity.EntityService.getEnvelopedAspects(EntityService.java:1504)
	at com.linkedin.metadata.entity.EntityService.getCorrespondingAspects(EntityService.java:353)
	at com.linkedin.metadata.entity.EntityService.getLatestEnvelopedAspects(EntityService.java:307)
	at com.linkedin.metadata.entity.EntityService.getEntitiesV2(EntityService.java:263)
	at com.linkedin.entity.client.JavaEntityClient.batchGetV2(JavaEntityClient.java:103)
	at com.datahub.authorization.PolicyFetcher.fetchPolicies(PolicyFetcher.java:59)
	at com.datahub.authorization.PolicyFetcher.fetchPolicies(PolicyFetcher.java:42)
	at com.datahub.authorization.DataHubAuthorizer$PolicyRefreshRunnable.run(DataHubAuthorizer.java:222)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
Caused by: java.sql.SQLNonTransientConnectionException: Could not connect to address=(host=<redacted>)(port=3306)(type=master) : Could not connect to <redacted>:3306 : PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
	at org.mariadb.jdbc.internal.util.exceptions.ExceptionFactory.createException(ExceptionFactory.java:73)
	at org.mariadb.jdbc.internal.util.exceptions.ExceptionFactory.create(ExceptionFactory.java:192)
	at org.mariadb.jdbc.internal.protocol.AbstractConnectProtocol.connectWithoutProxy(AbstractConnectProtocol.java:1346)
	at org.mariadb.jdbc.internal.util.Utils.retrieveProxy(Utils.java:634)
	at org.mariadb.jdbc.MariaDbConnection.newConnection(MariaDbConnection.java:150)
	at org.mariadb.jdbc.Driver.connect(Driver.java:89)
	at java.sql.DriverManager.getConnection(DriverManager.java:664)
	at java.sql.DriverManager.getConnection(DriverManager.java:208)
	at io.ebean.datasource.pool.ConnectionPool.createUnpooledConnection(ConnectionPool.java:529)
	at io.ebean.datasource.pool.ConnectionPool.createUnpooledConnection(ConnectionPool.java:524)
	at io.ebean.datasource.pool.ConnectionPool.createConnectionForQueue(ConnectionPool.java:766)
	at io.ebean.datasource.pool.PooledConnectionQueue._getPooledConnection(PooledConnectionQueue.java:314)
	at io.ebean.datasource.pool.PooledConnectionQueue.getPooledConnection(PooledConnectionQueue.java:270)
	at io.ebean.datasource.pool.ConnectionPool.getPooledConnection(ConnectionPool.java:817)
	at io.ebean.datasource.pool.ConnectionPool.getConnection(ConnectionPool.java:805)
	at io.ebeaninternal.server.transaction.TransactionFactoryBasic.createQueryTransaction(TransactionFactoryBasic.java:28)
	... 24 common frames omitted
I deleted gms pod. It was restarted. And it is able to connect to database again. I wonder what happened earlier