https://linen.dev logo
Join Slack
Powered by
This message was deleted.
# troubleshooting
s
This message was deleted.
t
Last starting log : 
2022-12-09T14:35:27,100 INFO [main] org.apache.druid.sql.calcite.schema.SegmentMetadataCache - SegmentMetadataCache initialized in [755,644] ms.
Running druid 24.0.1
and 
druid.sql.planner.metadataSegmentCacheEnable
is set to False (default value)
d
How many segment files do you have?
t
Around 37k
But I don’t want broker to load segments, only segment metadata
k
Broker only loads the segment metadata always. I find it suspicious that broker is taking 12 mins just to initialise.
Copy code
|`druid.broker.segment.awaitInitializationOnStart`|Boolean|Whether the Broker will wait for its view of segments to fully initialize before starting up. If set to 'true', the Broker's HTTP server will not start up, and the Broker will not announce itself as available, until the server view is initialized. See also `druid.sql.planner.awaitInitializationOnStart`, a related setting.|true|
You can set this property to false. The downside is that broker might give incorrect results while the whole metadata is loaded.
Do you see exceptions like 
Metadata refresh failed, trying again soon.
in the logs ?
Are other services also restarted at the same time ?
t
Yep, I do see those exceptions
k
Are other services also restarted at the same time ?
t
Apart from other brokers, no
k
We would be logging the exception. Those exception stack traces would point to what is going on.
t
Okay, thank you Karan, gonna check
Here is what I found in 
/var/log/syslog/
:
Copy code
Dec 12 15:40:12 druid-query run-druid[29103]: Exception in thread "BrokerServerView-5" java.util.concurrent.RejectedExecutionException: Task org.apache.druid.client.BrokerServerView$$Lambda$307/624230821@5f2810c9 rejected from java.util.concurrent.ThreadPoolExecutor@5717b890[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 44035]
Dec 12 15:40:12 druid-query run-druid[29103]: #011at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063)
Dec 12 15:40:12 druid-query run-druid[29103]: #011at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830)
Dec 12 15:40:12 druid-query run-druid[29103]: #011at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379)
Dec 12 15:40:12 druid-query run-druid[29103]: #011at java.util.concurrent.Executors$DelegatedExecutorService.execute(Executors.java:668)
Dec 12 15:40:12 druid-query run-druid[29103]: #011at org.apache.druid.client.BrokerServerView.runTimelineCallbacks(BrokerServerView.java:397)
Dec 12 15:40:12 druid-query run-druid[29103]: #011at org.apache.druid.client.BrokerServerView.serverAddedSegment(BrokerServerView.java:293)
Dec 12 15:40:12 druid-query run-druid[29103]: #011at org.apache.druid.client.BrokerServerView.access$000(BrokerServerView.java:67)
Dec 12 15:40:12 druid-query run-druid[29103]: #011at org.apache.druid.client.BrokerServerView$1.segmentAdded(BrokerServerView.java:149)
Dec 12 15:40:12 druid-query run-druid[29103]: #011at org.apache.druid.client.FilteringSegmentCallback.segmentAdded(FilteringSegmentCallback.java:52)
Dec 12 15:40:12 druid-query run-druid[29103]: #011at org.apache.druid.client.BatchServerInventoryView.lambda$addSingleInventory$2(BatchServerInventoryView.java:278)
Dec 12 15:40:12 druid-query run-druid[29103]: #011at org.apache.druid.client.BatchServerInventoryView.lambda$runSegmentCallbacks$0(BatchServerInventoryView.java:239)
Dec 12 15:40:12 druid-query run-druid[29103]: #011at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
Dec 12 15:40:12 druid-query run-druid[29103]: #011at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
Dec 12 15:40:12 druid-query run-druid[29103]: #011at java.lang.Thread.run(Thread.java:750)
k
This might look like a concurrency issue. Would need to full broker log to confirm . I would suggest creating a github ticket with the broker logs so we can look at it
Copy code
druid.serverview.http.numThreads
Default value is 5.
You could increase it to 15 and see if there is any improvement
t
Alright thanks Karan, will create a Github issue soon
d
Weird that we never have this problem. But by default we set http.numThreads = 500 everywhere.
k
Didip these are not the netty threads which you might set to such a big number
These are the threads which go to each historical and get the segments served by that historical so that the broker can plan stuff
and decide where to send the query
t
Hello, after running some queries on a large period, my brokers crashed, but it seems that the query was still running on my historicals. Once I saw that my brokers have crashed, I’ve restarted them and it took a long time. But I’ve noticed that once the CPU usage on historicals dropped, the brokers started. It looks like the fact that historicals were busy, brokers could not start
Open in Slack
PreviousNext
Powered by