This message was deleted Apache Druid #general

Join Slack
This message was deleted.
# general
Slackbot
08/30/2023, 6:34 AM
This message was deleted.
Arun C
08/30/2023, 12:54 PM
Is the coordinator/overlord OOMing when you said it's restarting?
Arun C
08/30/2023, 12:54 PM
Also what version of druid are you on?
sriramdas sivasai
08/30/2023, 5:03 PM
Im using v25 and cluster utilization is at 70%. sorry coordinator is not restarting, its being unhealthy. because of which there is less number of tasks being submitted/running state.
sriramdas sivasai
08/30/2023, 9:18 PM
Copy code
java.util.concurrent.TimeoutException: Timeout waiting for task.
at com.google.common.util.concurrent.AbstractFuture$Sync.get(AbstractFuture.java:269) ~[guava-16.0.1.jar:?]
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:96) ~[guava-16.0.1.jar:?]
at org.apache.druid.indexing.seekablestream.supervisor.SeekableStreamSupervisor.coalesceAndAwait(SeekableStreamSupervisor.java:4126) ~[druid-indexing-service-25.0.0.jar:25.0.0]
at org.apache.druid.indexing.seekablestream.supervisor.SeekableStreamSupervisor.updateCurrentOffsets(SeekableStreamSupervisor.java:3771) ~[druid-indexing-service-25.0.0.jar:25.0.0]
at org.apache.druid.indexing.seekablestream.supervisor.SeekableStreamSupervisor.updateCurrentAndLatestOffsets(SeekableStreamSupervisor.java:3738) ~[druid-indexing-service-25.0.0.jar:25.0.0]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) ~[?:?]
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) ~[?:?]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
at java.lang.Thread.run(Thread.java:829) ~[?:?]
2023-08-30T20:43:35,587 WARN [IndexTaskClient-data5g-1] org.apache.druid.java.util.common.RetryUtils - Retrying (1 of 9) in 1,196ms.
org.skife.jdbi.v2.exceptions.UnableToObtainConnectionException: java.sql.SQLException: Cannot get a connection, general error
at org.skife.jdbi.v2.DBI.open(DBI.java:230) ~[jdbi-2.63.1.jar:2.63.1]
at org.skife.jdbi.v2.DBI.withHandle(DBI.java:279) ~[jdbi-2.63.1.jar:2.63.1]
at org.apache.druid.metadata.SQLMetadataConnector.lambda$retryWithHandle$0(SQLMetadataConnector.java:142) ~[druid-server-25.0.0.jar:25.0.0]
at org.apache.druid.java.util.common.RetryUtils.retry(RetryUtils.java:129) ~[druid-core-25.0.0.jar:25.0.0]
at org.apache.druid.java.util.common.RetryUtils.retry(RetryUtils.java:81) ~[druid-core-25.0.0.jar:25.0.0]
at org.apache.druid.java.util.common.RetryUtils.retry(RetryUtils.java:163) ~[druid-core-25.0.0.jar:25.0.0]
at org.apache.druid.java.util.common.RetryUtils.retry(RetryUtils.java:153) ~[druid-core-25.0.0.jar:25.0.0]
at org.apache.druid.metadata.SQLMetadataConnector.retryWithHandle(SQLMetadataConnector.java:142) ~[druid-server-25.0.0.jar:25.0.0]
at org.apache.druid.metadata.SQLMetadataConnector.retryWithHandle(SQLMetadataConnector.java:152) ~[druid-server-25.0.0.jar:25.0.0]
at org.apache.druid.metadata.SQLMetadataStorageActionHandler.getStatus(SQLMetadataStorageActionHandler.java:257) ~[druid-server-25.0.0.jar:25.0.0]
at org.apache.druid.indexing.overlord.MetadataTaskStorage.getStatus(MetadataTaskStorage.java:191) ~[druid-indexing-service-25.0.0.jar:25.0.0]
at org.apache.druid.indexing.seekablestream.supervisor.SeekableStreamSupervisor$2.getTaskStatus(SeekableStreamSupervisor.java:862) ~[druid-indexing-service-25.0.0.jar:25.0.0]
at org.apache.druid.indexing.common.IndexTaskClient.submitRequest(IndexTaskClient.java:322) ~[druid-indexing-service-25.0.0.jar:25.0.0]
at org.apache.druid.indexing.common.IndexTaskClient.submitRequestWithEmptyContent(IndexTaskClient.java:219) ~[druid-indexing-service-25.0.0.jar:25.0.0]
at org.apache.druid.indexing.seekablestream.SeekableStreamIndexTaskClientSyncImpl.getCurrentOffsets(SeekableStreamIndexTaskClientSyncImpl.java:279) ~[druid-indexing-service-25.0.0.jar:25.0.0]
at org.apache.druid.indexing.seekablestream.SeekableStreamIndexTaskClientSyncImpl.lambda$getCurrentOffsetsAsync$6(SeekableStreamIndexTaskClientSyncImpl.java:407) ~[druid-indexing-service-25.0.0.jar:25.0.0]
at java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
at java.lang.Thread.run(Thread.java:829) ~[?:?]
Caused by: java.sql.SQLException: Cannot get a connection, general error
at org.apache.commons.dbcp2.PoolingDataSource.getConnection(PoolingDataSource.java:131) ~[commons-dbcp2-2.0.1.jar:2.0.1]
at org.apache.commons.dbcp2.BasicDataSource.getConnection(BasicDataSource.java:1413) ~[commons-dbcp2-2.0.1.jar:2.0.1]
at org.skife.jdbi.v2.DataSourceConnectionFactory.openConnection(DataSourceConnectionFactory.java:36) ~[jdbi-2.63.1.jar:2.63.1]
at org.skife.jdbi.v2.DBI.open(DBI.java:212) ~[jdbi-2.63.1.jar:2.63.1]
... 19 more
Caused by: java.lang.InterruptedException
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2056) ~[?:?]
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2090) ~[?:?]
at org.apache.commons.pool2.impl.LinkedBlockingDeque.takeFirst(LinkedBlockingDeque.java:524) ~[commons-pool2-2.2.jar:2.2]
at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:438) ~[commons-pool2-2.2.jar:2.2]
at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:361) ~[commons-pool2-2.2.jar:2.2]
at org.apache.commons.dbcp2.PoolingDataSource.getConnection(PoolingDataSource.java:119) ~[commons-dbcp2-2.0.1.jar:2.0.1]
at org.apache.commons.dbcp2.BasicDataSource.getConnection(BasicDataSource.java:1413) ~[commons-dbcp2-2.0.1.jar:2.0.1]
at org.skife.jdbi.v2.DataSourceConnectionFactory.openConnection(DataSourceConnectionFactory.java:36) ~[jdbi-2.63.1.jar:2.63.1]
at org.skife.jdbi.v2.DBI.open(DBI.java:212) ~[jdbi-2.63.1.jar:2.63.1]
... 19 more
2023-08-30T20:43:35,587 WARN [IndexTaskClient-data5g-2] org.apache.druid.java.util.common.RetryUtils - Retrying (1 of 9) in 1,240ms.
org.skife.jdbi.v2.exceptions.UnableToObtainConnectionException: java.sql.SQLException: Cannot get a connection, general error
at org.skife.jdbi.v2.DBI.open(DBI.java:230) ~[jdbi-2.63.1.jar:2.63.1]
at org.skife.jdbi.v2.DBI.withHandle(DBI.java:279) ~[jdbi-2.63.1.jar:2.63.1]
at org.apache.druid.metadata.SQLMetadataConnector.lambda$retryWithHandle$0(SQLMetadataConnector.java:142) ~[druid-server-25.0.0.jar:25.0.0]
at org.apache.druid.java.util.common.RetryUtils.retry(RetryUtils.java:129) ~[druid-core-25.0.0.jar:25.0.0]
at org.apache.druid.java.util.common.RetryUtils.retry(RetryUtils.java:81) ~[druid-core-25.0.0.jar:25.0.0]
at org.apache.druid.java.util.common.RetryUtils.retry(RetryUtils.java:163) ~[druid-core-25.0.0.jar:25.0.0]
at org.apache.druid.java.util.common.RetryUtils.retry(RetryUtils.java:153) ~[druid-core-25.0.0.jar:25.0.0]
at org.apache.druid.metadata.SQLMetadataConnector.retryWithHandle(SQLMetadataConnector.java:142) ~[druid-server-25.0.0.jar:25.0.0]
at org.apache.druid.metadata.SQLMetadataConnector.retryWithHandle(SQLMetadataConnector.java:152) ~[druid-server-25.0.0.jar:25.0.0]
at org.apache.druid.metadata.SQLMetadataStorageActionHandler.getStatus(SQLMetadataStorageActionHandler.java:257) ~[druid-server-25.0.0.jar:25.0.0]
at org.apache.druid.indexing.overlord.MetadataTaskStorage.getStatus(MetadataTaskStorage.java:191) ~[druid-indexing-service-25.0.0.jar:25.0.0]
at org.apache.druid.indexing.seekablestream.supervisor.SeekableStreamSupervisor$2.getTaskStatus(SeekableStreamSupervisor.java:862) ~[druid-indexing-service-25.0.0.jar:25.0.0]
at org.apache.druid.indexing.common.IndexTaskClient.submitRequest(IndexTaskClient.java:322) ~[druid-indexing-service-25.0.0.jar:25.0.0]
at org.apache.druid.indexing.common.IndexTaskClient.submitRequestWithEmptyContent(IndexTaskClient.java:219) ~[druid-indexing-service-25.0.0.jar:25.0.0]
at org.apache.druid.indexing.seekablestream.SeekableStreamIndexTaskClientSyncImpl.getCurrentOffsets(SeekableStreamIndexTaskClientSyncImpl.java:279) ~[druid-indexing-service-25.0.0.jar:25.0.0]
at org.apache.druid.indexing.seekablestream.SeekableStreamIndexTaskClientSyncImpl.lambda$getCurrentOffsetsAsync$6(SeekableStreamIndexTaskClientSyncImpl.java:407) ~[druid-indexing-service-25.0.0.jar:25.0.0]
at java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
at java.lang.Thread.run(Thread.java:829) ~[?:?]
Caused by: java.sql.SQLException: Cannot get a connection, general error
at org.apache.commons.dbcp2.PoolingDataSource.getConnection(PoolingDataSource.java:131) ~[commons-dbcp2-2.0.1.jar:2.0.1]
at org.apache.commons.dbcp2.BasicDataSource.getConnection(BasicDataSource.java:1413) ~[commons-dbcp2-2.0.1.jar:2.0.1]
at org.skife.jdbi.v2.DataSourceConnectionFactory.openConnection(DataSourceConnectionFactory.java:36) ~[jdbi-2.63.1.jar:2.63.1]
at org.skife.jdbi.v2.DBI.open(DBI.java:212) ~[jdbi-2.63.1.jar:2.63.1]
... 19 more
Caused by: java.lang.InterruptedException
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2056) ~[?:?]
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2090) ~[?:?]
at org.apache.commons.pool2.impl.LinkedBlockingDeque.takeFirst(LinkedBlockingDeque.java:524) ~[commons-pool2-2.2.jar:2.2]
at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:438) ~[commons-pool2-2.2.jar:2.2]
at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:361) ~[commons-pool2-2.2.jar:2.2]
at org.apache.commons.dbcp2.PoolingDataSource.getConnection(PoolingDataSource.java:119) ~[commons-dbcp2-2.0.1.jar:2.0.1]
at org.apache.commons.dbcp2.BasicDataSource.getConnection(BasicDataSource.java:1413) ~[commons-dbcp2-2.0.1.jar:2.0.1]
at org.skife.jdbi.v2.DataSourceConnectionFactory.openConnection(DataSourceConnectionFactory.java:36) ~[jdbi-2.63.1.jar:2.63.1]
at org.skife.jdbi.v2.DBI.open(DBI.java:212) ~[jdbi-2.63.1.jar:2.63.1]
... 19 more
above is the error im getting in coordinator while its trying to communicate with mysql i guess
sriramdas sivasai
08/30/2023, 9:18 PM
@Arun C
Arun C
08/30/2023, 11:04 PM
Seems to be the case. Is this something that recently occurred or is this your first setup
sriramdas sivasai
08/30/2023, 11:14 PM
@Arun C we are seeing now too even after reverting
Open in Slack
Previous Next