https://linen.dev logo
Join Slack
Powered by
This message was deleted.
# troubleshooting
s
This message was deleted.
b
Copy code
2022-12-23T00:23:17+00:00	middleManager	{"Time":"1970-01-01T00:00:00Z","Severity":"INFO","Component":"middleManager-ip-172-31-4-225","Message":"[pool-6-thread-1] org.apache.druid.emitter.kafka.KafkaEmitter - Exception while getting record from queue or producer send, Events would not be emitted anymore.: \nat java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2014) ~ [?:1.8.0_275]\nat java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2048) ~ [?:1.8.0_275]\nat java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) ~ [?:1.8.0_275]\nat org.apache.druid.emitter.kafka.MemoryBoundLinkedBlockingQueue.take(MemoryBoundLinkedBlockingQueue.java:59) ~ [?:?]\nat org.apache.druid.emitter.kafka.KafkaEmitter.sendToKafka(KafkaEmitter.java:156) ~ [?:?]\nat org.apache.druid.emitter.kafka.KafkaEmitter.sendMetricToKafka(KafkaEmitter.java:138) ~ [?:?]\nat java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~ [?:1.8.0_275]\nat java.util.concurrent.FutureTask.run(FutureTask.java:266) ~ [?:1.8.0_275]\nat java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) ~ [?:1.8.0_275]\nat java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) ~ [?:1.8.0_275]\nat java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~ [?:1.8.0_275]\nat java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~ [?:1.8.0_275]\nat java.lang.Thread.run(Thread.java:748) ~ [?:1.8.0_275]\n"}
2022-12-23T00:23:17+00:00	middleManager	{"Time":"1970-01-01T00:00:00Z","Severity":"INFO","Component":"middleManager-ip-172-31-4-225","Message":"[Curator-Framework-0] org.apache.curator.framework.imps.CuratorFrameworkImpl - backgroundOperationsLoop exiting: "}
2022-12-23T00:23:17+00:00	middleManager	{"Time":"1970-01-01T00:00:00Z","Severity":"INFO","Component":"middleManager-ip-172-31-4-225","Message":"[Thread-31] org.apache.zookeeper.ZooKeeper - Session: 0x100103961870005 closed: "}
2022-12-23T00:23:17+00:00	middleManager	{"Time":"1970-01-01T00:00:00Z","Severity":"INFO","Component":"middleManager-ip-172-31-4-225","Message":"[Thread-31] org.apache.druid.java.util.common.lifecycle.Lifecycle - Stopping lifecycle [module] stage [INIT]: "}
Full logs on middelManager bring up.
b
Although the release notes don't seem to mention a zk change from 0.23 -> 24.0.
b
Zookeeper version is > 3.4.10
b
I see kafka errors in the MM log, have to run an errand, bbiab
b
Since 0.23.0 is working fine, it is puzzling what is breaking as the backwards incompatible changes in the release notes have nothing relevant to the issue above.
b
true... Did you create new config files, or use the same?
b
Exact same configs as before
I will try with zookeeper version > 3.5 and update
b
Are you using 24.0.2, just in case there's a new bug fix that matters?
b
I have tried 24.0.0, .1 and .2 and same behavior
b
If it doesn't work w/3.5, a bug report would probably be good, even if we figure it out.
v
in Druid 0.22 support for ZK 3.4 was dropped < https://github.com/apache/druid/issues/11657#22-up-dropped-zk-3.4 I guess it would be nice if there was like a console warning message that told you about the bad ZK version. It is curious that stuff just happened to work.
Also, while you should definitely update you ZK to > 3.4 it should also be noted that in Druid 25 (which is currently in release candidate stage! https://lists.apache.org/thread/ry3w3778xx5lv6tdxtz2gh2fks2f6bn7) the defaults are changing to http based task announcement (see my response in this thread https://apachedruidworkspace.slack.com/archives/C0309C9L90D/p1671473660817489). I bet that switching away form using ZK for task announcement would help in this case also.
b
Confirming that zookeeper version is 3.5.9.
v
Can you paste the overlord log here?
b
logs:
v
is this the full log? I don’t see the message which states starting lifecycle
I can see below in your middle manager log
Copy code
Node[<http://192.168.5.232:8081>] of role[overlord] went offline.: "}
so the overlord is shutting down.
b
@Vijay Narayanan Yes, above are complete logs. What else I can do to troubleshooting why overload is shutting down?
v
what is the Xmx in the jvm config for the overlord. One possibility is that the overlord is running out of resources. You are running coordinator as overlord?
6 Views
Open in Slack
PreviousNext
Powered by