https://datahubproject.io logo
Join Slack
Powered by
I enabled airflow ingestion, and we have over 400+...
# troubleshoot
s
I enabled airflow ingestion, and we have over 400+ airflow DAGs and thousands of task run per day. From the gms log, i can see 20+ ingestion every second. The consequence is, i obverse search, view on datahub UI extremely slow sometimes. And find below errors from gms log. I would ask suggestions from experts how can i optimize? thank you.
Copy code
02:24:10.411 [Thread-5879] ERROR c.l.d.g.e.DataHubDataFetcherExceptionHandler:21 - Failed to execute DataFetcher
java.util.concurrent.CompletionException: java.lang.RuntimeException: Failed to retrieve entities of type DataFlow
	at java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:273)
	at java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:280)
	at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1606)
	at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.RuntimeException: Failed to retrieve entities of type DataFlow
	at com.linkedin.datahub.graphql.GmsGraphQLEngine.lambda$null$153(GmsGraphQLEngine.java:1376)
	at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1604)
	... 1 common frames omitted
Caused by: java.lang.RuntimeException: Failed to batch load Data Flows
	at com.linkedin.datahub.graphql.types.dataflow.DataFlowType.batchLoad(DataFlowType.java:118)
	at com.linkedin.datahub.graphql.GmsGraphQLEngine.lambda$null$153(GmsGraphQLEngine.java:1373)
	... 2 common frames omitted
Caused by: javax.persistence.PersistenceException: java.sql.SQLException: Unsuccessfully waited [1000] millis for a connection to be returned. No connections are free. You need to Increase the max connections of [50] or look for a connection pool leak using datasource.xxx.capturestacktrace=true
	at io.ebeaninternal.server.transaction.TransactionFactoryBasic.createQueryTransaction(TransactionFactoryBasic.java:35)
	at io.ebeaninternal.server.transaction.TransactionManager.createQueryTransaction(TransactionManager.java:360)
	at io.ebeaninternal.server.core.DefaultServer.createQueryTransaction(DefaultServer.java:2306)
	at io.ebeaninternal.server.core.OrmQueryRequest.initTransIfRequired(OrmQueryRequest.java:282)
	at io.ebeaninternal.server.core.DefaultServer.findList(DefaultServer.java:1595)
	at io.ebeaninternal.server.core.DefaultServer.findList(DefaultServer.java:1574)
	at io.ebeaninternal.server.querydefn.DefaultOrmQuery.findList(DefaultOrmQuery.java:1481)
	at com.linkedin.metadata.entity.ebean.EbeanAspectDao.batchGetUnion(EbeanAspectDao.java:324)
	at com.linkedin.metadata.entity.ebean.EbeanAspectDao.batchGet(EbeanAspectDao.java:244)
	at com.linkedin.metadata.entity.ebean.EbeanAspectDao.batchGet(EbeanAspectDao.java:225)
	at com.linkedin.metadata.entity.ebean.EbeanEntityService.getEnvelopedAspects(EbeanEntityService.java:710)
	at com.linkedin.metadata.entity.ebean.EbeanEntityService.getLatestEnvelopedAspects(EbeanEntityService.java:219)
	at com.linkedin.metadata.entity.EntityService.getEntitiesV2(EntityService.java:175)
	at com.linkedin.entity.client.JavaEntityClient.batchGetV2(JavaEntityClient.java:98)
	at com.linkedin.datahub.graphql.types.dataflow.DataFlowType.batchLoad(DataFlowType.java:102)
	... 3 common frames omitted
Caused by: java.sql.SQLException: Unsuccessfully waited [1000] millis for a connection to be returned. No connections are free. You need to Increase the max connections of [50] or look for a connection pool leak using datasource.xxx.capturestacktrace=true
	at io.ebean.datasource.pool.PooledConnectionQueue._getPooledConnectionWaitLoop(PooledConnectionQueue.java:356)
	at io.ebean.datasource.pool.PooledConnectionQueue._getPooledConnection(PooledConnectionQueue.java:330)
	at io.ebean.datasource.pool.PooledConnectionQueue.getPooledConnection(PooledConnectionQueue.java:270)
	at io.ebean.datasource.pool.ConnectionPool.getPooledConnection(ConnectionPool.java:817)
	at io.ebean.datasource.pool.ConnectionPool.getConnection(ConnectionPool.java:805)
	at io.ebeaninternal.server.transaction.TransactionFactoryBasic.createQueryTransaction(TransactionFactoryBasic.java:28)
	... 17 common frames omitted
b
Hi there! This typically happens when the connection pool to the document store (e.g. mysql) is filled up by requests to read or write. One way to address this would be to increase the max number of connections (if you truly will have this level of volume) you can have active to your DB by setting the 
EBEAN_MAX_CONNECTIONS
environment variable in the 
datahub-gms
pod to > 50. However, you're DB must support this connection load so the first thing would be to verify your key-value DB is scaled up enough
thank you 1
3 Views
Open in Slack
PreviousNext
Powered by