Slackbot
03/16/2023, 8:57 AMSlackbot
03/16/2023, 11:11 AMSlackbot
03/16/2023, 2:38 PMSlackbot
03/16/2023, 10:39 PMSlackbot
03/17/2023, 6:28 AMSlackbot
03/17/2023, 6:57 AMSlackbot
03/17/2023, 8:24 AMSlackbot
03/17/2023, 2:24 PMRenato Santos
03/17/2023, 6:13 PM2023-03-17T18:07:48,636 INFO [parent-monitor-0] org.apache.druid.indexing.worker.executor.ExecutorLifecycle - Triggering JVM shutdown.
and then the tasks stop but the status is still the same
2023-03-17T18:07:49,024 INFO [Thread-69] org.apache.druid.java.util.common.lifecycle.Lifecycle - Stopping lifecycle [module] stage [INIT]
Maybe it's k8s sending a kill signal, I've increase the memory to 12gb per MM, and i'm using -Xms512M -Xmx1322M -XX:MaxDirectMemorySize=3g
for the peon taskSlackbot
03/17/2023, 6:53 PMSlackbot
03/20/2023, 12:24 PMSlackbot
03/20/2023, 1:20 PMMartin Maqueira
03/20/2023, 1:21 PM{
"id": "index_kafka_qps_e1e08c491120d95_dkgbobjj",
"groupId": "index_kafka_qps",
"type": "index_kafka",
"createdTime": "2023-03-20T12:12:49.298Z",
"queueInsertionTime": "1970-01-01T00:00:00.000Z",
"statusCode": "FAILED",
"status": "FAILED",
"runnerStatusCode": "WAITING",
"duration": -1,
"location": {
"host": "172.31.13.78",
"port": 8103,
"tlsPort": -1
},
"dataSource": "qps",
"errorMsg": "No task in the corresponding pending completion taskGroup[0] succeeded before completion timeout ela..."
}
Martin Maqueira
03/20/2023, 1:25 PMMartin Maqueira
03/20/2023, 1:26 PM2023-03-20T12:18:05,279 INFO [Thread-70] org.apache.druid.curator.announcement.Announcer - Unannouncing [/druid/announcements/172.31.13.78:8103]
2023-03-20T12:18:05,329 INFO [Thread-70] org.apache.druid.curator.announcement.Announcer - Unannouncing [/druid/segments/172.31.13.78:8103/172.31.13.78:8103_indexer-executor__default_tier_2023-03-20T12:13:04.001Z_c69a5170b7ee404b8bc71df4625354e10]
2023-03-20T12:18:05,338 INFO [Thread-70] org.apache.druid.curator.announcement.Announcer - Unannouncing [/druid/internal-discovery/PEON/172.31.13.78:8103]
2023-03-20T12:18:05,346 INFO [Thread-70] org.apache.druid.java.util.common.lifecycle.Lifecycle - Stopping lifecycle [module] stage [SERVER]
2023-03-20T12:18:05,373 INFO [Thread-70] org.eclipse.jetty.server.AbstractConnector - Stopped ServerConnector@c6c84fa{HTTP/1.1, (http/1.1)}{0.0.0.0:8103}
2023-03-20T12:18:05,374 INFO [Thread-70] org.eclipse.jetty.server.session - node0 Stopped scavenging
2023-03-20T12:18:05,380 INFO [Thread-70] org.eclipse.jetty.server.handler.ContextHandler - Stopped o.e.j.s.ServletContextHandler@5e05a706{/,null,STOPPED}
2023-03-20T12:18:05,391 INFO [Thread-70] org.apache.druid.java.util.common.lifecycle.Lifecycle - Stopping lifecycle [module] stage [NORMAL]
2023-03-20T12:18:05,395 INFO [Thread-70] org.apache.druid.server.coordination.ZkCoordinator - Stopping ZkCoordinator for [DruidServerMetadata{name='172.31.13.78:8103', hostAndPort='172.31.13.78:8103', hostAndTlsPort='null', maxSize=0, tier='_default_tier', type=indexer-executor, priority=0}]
2023-03-20T12:18:05,395 INFO [Thread-70] org.apache.druid.server.coordination.SegmentLoadDropHandler - Stopping...
2023-03-20T12:18:05,395 INFO [Thread-70] org.apache.druid.server.coordination.SegmentLoadDropHandler - Stopped.
2023-03-20T12:18:05,395 INFO [Thread-70] org.apache.druid.indexing.overlord.SingleTaskBackgroundRunner - Starting graceful shutdown of task[index_kafka_qps_e1e08c491120d95_dkgbobjj].
2023-03-20T12:18:05,396 INFO [Thread-70] org.apache.druid.indexing.seekablestream.SeekableStreamIndexTaskRunner - Stopping forcefully (status: [READING])
2023-03-20T12:18:05,398 ERROR [task-runner-0-priority-0] org.apache.druid.indexing.seekablestream.SeekableStreamIndexTaskRunner - Encountered exception in run() before persisting.
org.apache.kafka.common.errors.InterruptException: java.lang.InterruptedException
at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.maybeThrowInterruptException(ConsumerNetworkClient.java:520) ~[?:?]
at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:281) ~[?:?]
at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:236) ~[?:?]
at org.apache.kafka.clients.consumer.KafkaConsumer.pollForFetches(KafkaConsumer.java:1297) ~[?:?]
at org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1238) ~[?:?]
at org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1211) ~[?:?]
at org.apache.druid.indexing.kafka.KafkaRecordSupplier.poll(KafkaRecordSupplier.java:128) ~[?:?]
at org.apache.druid.indexing.kafka.IncrementalPublishingKafkaIndexTaskRunner.getRecords(IncrementalPublishingKafkaIndexTaskRunner.java:95) ~[?:?]
at org.apache.druid.indexing.seekablestream.SeekableStreamIndexTaskRunner.runInternal(SeekableStreamIndexTaskRunner.java:612) [druid-indexing-service-0.23.0.jar:0.23.0]
at org.apache.druid.indexing.seekablestream.SeekableStreamIndexTaskRunner.run(SeekableStreamIndexTaskRunner.java:265) [druid-indexing-service-0.23.0.jar:0.23.0]
at org.apache.druid.indexing.seekablestream.SeekableStreamIndexTask.run(SeekableStreamIndexTask.java:149) [druid-indexing-service-0.23.0.jar:0.23.0]
at org.apache.druid.indexing.overlord.SingleTaskBackgroundRunner$SingleTaskBackgroundRunnerCallable.call(SingleTaskBackgroundRunner.java:477) [druid-indexing-service-0.23.0.jar:0.23.0]
at org.apache.druid.indexing.overlord.SingleTaskBackgroundRunner$SingleTaskBackgroundRunnerCallable.call(SingleTaskBackgroundRunner.java:449) [druid-indexing-service-0.23.0.jar:0.23.0]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_275]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_275]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_275]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_275]
Caused by: java.lang.InterruptedException
... 17 more
Martin Maqueira
03/20/2023, 1:28 PMMartin Maqueira
03/20/2023, 1:28 PMMartin Maqueira
03/20/2023, 1:40 PMMartin Maqueira
03/20/2023, 1:41 PMVijay Narayanan
03/20/2023, 1:59 PMSlackbot
03/20/2023, 2:01 PMSlackbot
03/20/2023, 2:06 PMLuiz Augusto
03/20/2023, 2:07 PMSlackbot
03/20/2023, 2:27 PMSlackbot
03/20/2023, 7:57 PMSlackbot
03/21/2023, 8:47 AMSlackbot
03/21/2023, 2:37 PMSlackbot
03/22/2023, 12:37 AMSlackbot
03/22/2023, 9:53 AMSlackbot
03/22/2023, 12:05 PM