This message was deleted.

Could you be overextending the resources when tasks switch to publishing mode? What do CPU and memory utilization look like on the nodes when this happens? What CPU and Memory resources do you have on the nodes? And how are they split amongst the Peons? In other words, what are the settings for javaOptsArray for the Peons in the Middle Manager config?

My MiddleManager has 8 vcpu and 64 GB RAM. CPU utilisation is around 70%-85% during simultaneous reading and publishing phase. Memory utilisation is around 40GB-50GB during simultaneous reading and publishing phase Split amongst the Peons :

druid.indexer.runner.javaOpts=-server -Xmx3g -Daws.region={{region}} -Duser.timezone=UTC -Dfile.encoding=UTF-8 -Djava.util.logging.manager=org.apache.logging.log4j.jul.LogManager -XX:+UseG1GC -XX:MaxGCPauseMillis=100 -XX:+PrintGCDetails -XX:+PrintGCTimeStamps

worker.capacity = 7 [1 cpu per worker] per node Total No of MM = 5 [35 workers in total]

Seems like you are good on resources, the only missing items I see are

-Xms3g

(recommended to be set the same as

-Xmx

) and

-XX:MaxDirectMemorySize=3g

set this based on

(druid.processing.numThreads + druid.processing.numMergeBuffers + 1) * druid.processing.buffer.sizeBytes

What version are you on? I believe you are hitting this issue: https://github.com/apache/druid/issues/11414 There are significant improvements on this included in 25.0: 1. https://github.com/apache/druid/pull/13328 -- Optimize the metadata calls 2. https://github.com/apache/druid/pull/13334 -- Do not queue more than one run notice at a time. 3. https://github.com/apache/druid/pull/13354 -- Contacts tasks asynchronously, eliminate chat threads and reduce the load on worker threads. 4. https://github.com/apache/druid/pull/13369 -- to speed up segment allocation and reduce metadata calls by clubbing the multiple requests together.

What about the network bandwidth. With tasks reading writing how much throughput would be needed on the network? How many GB/s would be required on the ingest to keep up with kafka?

i would also check if you have pending tasks while the publishing is happening