https://datahubproject.io logo
Join Slack
Powered by
Hello! Does someone know what can cause this error...
# troubleshoot
g
Hello! Does someone know what can cause this error on GMS instances?
Copy code
[gmsEbeanServiceConfig.heartBeat] ERROR i.e.datasource.pool.PooledConnection:311 - Error when fully closing connection [name[gmsEbeanServiceConfig16] slot[7] startTime[1664401306308] busySeconds[150] stackTrace[] stmt[null]]
java.sql.SQLNonTransientConnectionException: Communications link failure during rollback(). Transaction resolution unknown.
        at com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:110)
        at com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:97)
        at com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:89)
        at com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:63)
        at com.mysql.cj.jdbc.ConnectionImpl.rollback(ConnectionImpl.java:1848)
        at com.mysql.cj.jdbc.ConnectionImpl.realClose(ConnectionImpl.java:1705)
        at com.mysql.cj.jdbc.ConnectionImpl.close(ConnectionImpl.java:721)
        at io.ebean.datasource.pool.PooledConnection.closeConnectionFully(PooledConnection.java:308)
        at io.ebean.datasource.pool.FreeConnectionBuffer.trim(FreeConnectionBuffer.java:91)
        at io.ebean.datasource.pool.PooledConnectionQueue.trimInactiveConnections(PooledConnectionQueue.java:442)
        at io.ebean.datasource.pool.PooledConnectionQueue.trim(PooledConnectionQueue.java:422)
        at io.ebean.datasource.pool.ConnectionPool.trimIdleConnections(ConnectionPool.java:441)
        at io.ebean.datasource.pool.ConnectionPool.checkDataSource(ConnectionPool.java:459)
        at io.ebean.datasource.pool.ConnectionPool.access$000(ConnectionPool.java:43)
        at io.ebean.datasource.pool.ConnectionPool$HeartBeatRunnable.run(ConnectionPool.java:260)
        at java.util.TimerThread.mainLoop(Timer.java:555)
        at java.util.TimerThread.run(Timer.java:505)
b
hi, when are you seeing this error? first time installment or later?
g
This error i saw after i tried to search for some dataset and get an error message.
b
is your MySQL database available, can you see any errors in its logs?
g
Yes. The CPU consumption is low (10%) and the average of connections open is 60. I just see one error of idle sessions killed by MySQL. Let me get the mensage here to share
Example:
Copy code
[Note] Aborted connection 977989 to db: 'datahub-dev' user: 'datahub-dev' host: 'xxxxx' (Got timeout reading communication packets)
I found this: "I realize now that this error occurs when idle threads are killed by wait_timeout" https://dba.stackexchange.com/questions/205398/mysql-got-timeout-reading-communication-packets-when-reading-federated-table
Another common error:
Copy code
Bad handshake
"Bad handshake error means that server's handshake can't reach the client"
Let me share some informations about MySQL instance and eben configuration: Database version is MySQL 5.7.38 Eben config:
Copy code
ebean:
      # -- Connection wait timeout
      waitTimeoutMillis: 30000
      # -- Max connections
      maxConnections: 50
      # -- Min connections
      minConnections: 4
      # -- Max connection idle time in seconds
      maxInactiveTimeSeconds: 120
      # -- Max connection age in minutes
      maxAgeMinutes: 120
      # -- Leak time in minutes
      leakTimeMinutes: 15
b
is the mysql part of helm install or quickstart or an instance hosted?
g
Is a cloud SQL instance (GCP)
I'm using jdbc driver with this url params:
Copy code
?verifyServerCertificate=false&useSSL=true&useUnicode=yes&characterEncoding=UTF-8&enabledTLSProtocols=TLSv1.2
Copy code
[pool-11-thread-1] ERROR i.e.datasource.pool.PooledConnection:311 - Error when fully closing connection [name[gmsEbeanServiceConfig1291] slot[14] startTime[1664468657276] busySeconds[126] stackTrace[] stmt[select urn, aspect, version, metadata, systemMetadata, createdOn, createdBy, createdFor FROM metadata_aspect_v2 WHERE urn = ? AND aspect = ? AND version = ? UNION ALL SELECT urn, aspect, version, metadata, systemMetadata, createdOn, createdBy, createdFor FROM metadata_aspect_v2 WHERE urn = ? AND aspect = ? AND version = ? UNION ALL SELECT urn, aspect, version, metadata, systemMetadata, createdOn, createdBy, createdFor FROM metadata_aspect_v2 WHERE urn = ? AND aspect = ? AND version = ? UNION ALL SELECT urn, aspect, version, metadata, systemMetadata, createdOn, createdBy, createdFor FROM metadata_aspect_v2 WHERE urn = ? AND aspect = ? AND version = ? UNION ALL SELECT urn, aspect, version, metadata, systemMetadata, createdOn, createdBy, createdFor FROM metadata_aspect_v2 WHERE urn = ? AND aspect = ? AND version = ?]]
java.sql.SQLNonTransientConnectionException: Communications link failure during rollback(). Transaction resolution unknown.
	at com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:110)
	at com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:97)
	at com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:89)
	at com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:63)
	at com.mysql.cj.jdbc.ConnectionImpl.rollback(ConnectionImpl.java:1848)
	at com.mysql.cj.jdbc.ConnectionImpl.realClose(ConnectionImpl.java:1705)
	at com.mysql.cj.jdbc.ConnectionImpl.close(ConnectionImpl.java:721)
	at io.ebean.datasource.pool.PooledConnection.closeConnectionFully(PooledConnection.java:308)
	at io.ebean.datasource.pool.FreeConnectionBuffer.trim(FreeConnectionBuffer.java:91)
	at io.ebean.datasource.pool.PooledConnectionQueue.trimInactiveConnections(PooledConnectionQueue.java:442)
	at io.ebean.datasource.pool.PooledConnectionQueue.trim(PooledConnectionQueue.java:422)
	at io.ebean.datasource.pool.ConnectionPool.trimIdleConnections(ConnectionPool.java:441)
	at io.ebean.datasource.pool.ConnectionPool.checkDataSource(ConnectionPool.java:459)
	at io.ebean.datasource.pool.ConnectionPool.returnTheConnection(ConnectionPool.java:707)
	at io.ebean.datasource.pool.ConnectionPool.returnConnectionForceClose(ConnectionPool.java:691)
	at io.ebean.datasource.pool.PooledConnection.close(PooledConnection.java:481)
	at io.ebeaninternal.server.transaction.JdbcTransaction.deactivate(JdbcTransaction.java:925)
	at io.ebeaninternal.server.transaction.JdbcTransaction.commit(JdbcTransaction.java:1065)
	at io.ebeaninternal.server.core.OrmQueryRequest.endTransIfRequired(OrmQueryRequest.java:364)
	at io.ebeaninternal.server.core.DefaultServer.findList(DefaultServer.java:1599)
	at io.ebeaninternal.server.core.DefaultServer.findList(DefaultServer.java:1574)
	at io.ebeaninternal.server.querydefn.DefaultOrmQuery.findList(DefaultOrmQuery.java:1481)
	at com.linkedin.metadata.entity.ebean.EbeanAspectDao.batchGetUnion(EbeanAspectDao.java:359)
	at com.linkedin.metadata.entity.ebean.EbeanAspectDao.batchGet(EbeanAspectDao.java:279)
	at com.linkedin.metadata.entity.ebean.EbeanAspectDao.batchGet(EbeanAspectDao.java:260)
	at com.linkedin.metadata.entity.EntityService.exists(EntityService.java:1309)
	at com.linkedin.metadata.resources.entity.ResourceUtils.lambda$validateSearchResult$0(ResourceUtils.java:52)
	at java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:174)
	at java.util.Iterator.forEachRemaining(Iterator.java:116)
	at java.util.Spliterators$IteratorSpliterator.forEachRemaining(Spliterators.java:1801)
	at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482)
	at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:472)
	at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708)
	at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
	at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:566)
	at com.linkedin.metadata.resources.entity.ResourceUtils.validateSearchResult(ResourceUtils.java:53)
	at com.linkedin.entity.client.JavaEntityClient.search(JavaEntityClient.java:297)
	at com.datahub.authorization.PolicyFetcher.fetchPolicies(PolicyFetcher.java:50)
	at com.datahub.authorization.PolicyFetcher.fetchPolicies(PolicyFetcher.java:42)
	at com.datahub.authorization.DataHubAuthorizer$PolicyRefreshRunnable.run(DataHubAuthorizer.java:222)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
@dazzling-judge-80093 @big-carpet-38439 Do you have any idea?
Copy code
[pool-11-thread-1] ERROR c.d.authorization.DataHubAuthorizer:229 - Failed to retrieve policy urns! Skipping updating policy cache until next refresh. start: 0, count: 30
javax.persistence.PersistenceException: java.sql.SQLNonTransientConnectionException: No operations allowed after connection closed.
	at io.ebeaninternal.server.transaction.JdbcTransaction.doRollback(JdbcTransaction.java:1151)
	at io.ebeaninternal.server.transaction.JdbcTransaction.commit(JdbcTransaction.java:1061)
	at io.ebeaninternal.server.core.OrmQueryRequest.endTransIfRequired(OrmQueryRequest.java:364)
	at io.ebeaninternal.server.core.DefaultServer.findList(DefaultServer.java:1599)
	at io.ebeaninternal.server.core.DefaultServer.findList(DefaultServer.java:1574)
	at io.ebeaninternal.server.querydefn.DefaultOrmQuery.findList(DefaultOrmQuery.java:1481)
	at com.linkedin.metadata.entity.ebean.EbeanAspectDao.batchGetUnion(EbeanAspectDao.java:359)
	at com.linkedin.metadata.entity.ebean.EbeanAspectDao.batchGet(EbeanAspectDao.java:279)
	at com.linkedin.metadata.entity.ebean.EbeanAspectDao.batchGet(EbeanAspectDao.java:260)
	at com.linkedin.metadata.entity.EntityService.exists(EntityService.java:1309)
It seems that my GMS is having problems executing operations on my MySQL instance.
Because of these errors, I'm receiving a lot of "DataFetchingException" on the UI.
@bulky-soccer-26729 Any idea?
b
do other operations work fine? only fetching fails?
g
Yes. Create, edit and delete are working fine on UI.
I created a new SQL instance with MySQL 8+. Still having the same problems
• Server] Bad handshake • [Server] 'wait_timeout' period of 28800 seconds was exceeded for 
datahub-dev
• [Server] Aborted connection 939 to db: 'datahub-dev' user: 'datahub-dev'
A lot of these errors
Hello @bulky-electrician-72362 , I discovered what was going on! I'm using Google Cloud DNS service to serve Hostnames for my Cloud SQL instances. With that, I have the flexibility to change my domain backend easily. But, this service has a TTL configuration that I was using with the default values of 5 minutes. So, the Google DNS service was closing the GMS's database connections after this TTL period.
b
Ahh, thanks @gentle-camera-33498 for sharing this. This is really valuable info for the community.
g
Maybe another thing that could impact database connection: I removed the "useSSL" from the connection parameters. I'm using the following parameters:
Copy code
<host>?verifyServerCertificate=false&useUnicode=yes&characterEncoding=UTF-8&enabledTLSProtocols=TLSv1.2
https://github.com/acryldata/datahub-helm/blob/master/charts/datahub/subcharts/datahub-gms/values.yaml#L142
72 Views
Open in Slack
PreviousNext
Powered by