https://linen.dev logo
Join Slack
Powered by
This message was deleted.
# troubleshooting
s
This message was deleted.
s
You can control how many threads the historicals use to download segments with
Copy code
druid.segmentCache.numLoadingThreads
y
🤦 makes sense 🙂
s
Also, 
maxSegmentsInNodeLoadingQueue
 controls how many pending operations are given to each historical. Default of 100 should be plenty.
Let us know how it goes.
y
Same
Copy code
numLoadingThreads
to 5, and maxSegmentsInNodeLoadingQueue was already 200
s
Do you see pending loads on the historicals in the Druid Console?
y
Yes
s
hmmm what else could throttle the download?
just to confirm, did you restart the historicals after the change?
y
Yes I did
a
Is the location hosting the segments able to do ~25Gb/s? Outside of Druid are you able to get that kind of bandwidth a CLI tool?
y
Yes sir… even getting data via s3cli runs ~10Gb+
a
That includes writing and flushing to disk? Do you have a NVMe raid setup for Druid? I am assuming disk IO is the bottle neck here but maybe I am missing something.
g
IIRC, there's some additional config needed to actually parallelize loading beyond 
numLoadingThreads
— I believe there's also gates for feeding the threads
for http load queues (
druid.coordinator.loadqueuepeon.type=http
) IIRC you also want to set 
druid.coordinator.loadqueuepeon.http.batchSize=N
(where 
N <= numLoadingThreads
)
I don't remember how it works for ZK
druid.coordinator.loadqueuepeon.curator.numCallbackThreads
seems relevant from reading the docs, although i don't have personal experience with it
Mostly use HTTP these days 🙂
y
This was it my friend 🙂
Copy code
druid.coordinator.loadqueuepeon.type=http
druid.coordinator.loadqueuepeon.http.batchSize=50
🙌 1
g
nice! good to hear
Open in Slack
PreviousNext
Powered by