This message was deleted.

Besides updating those parameters to provide more resources to the query side, you can adjust how intermediate persists in the streaming ingestion works. The defaults are intended to focus more on ingestion throughput than the query side. How much data are you consuming? How is it partitioned on time? What is the duration of your tasks? Sometimes lowering the

intermediatePersistPeriod

or the

maxRowsInMemory

to smaller numbers can help the query side because the persisted files are in segment format which includes indexing. You don't want to go too low because it will cause too many persists though.

We are seeing that a spike / high qps causes ingestion lag. So was wondering if those setting may have caused this

For real-time ingestion there is a tradeoff between ingestion throughput and query performance. Larger intermediate persists will improve ingestion, but more frequent ones can improve query performance. The reason is that intermediate persists are created from the row buffers into Druid Segment format. Queries on the row buffers must scan all the rows to resolve, with the intermediate persists they can take advantage of the indexing that the segment format has. That said, you do not want to have too many intermediate persists, because too many will have overhead cost.

the peon query stack is basically the same design as the historical query stack, so it would be able to handle just as much QPS as a historical (i.e. a lot — and likely much more since it has much less data!!)

Thanks for both your help, @Gian Merlino @Sergio Ferragut Some update on what we tried: (Druid version 0.20) We are running MMs on r5.24xlarge (96 vCPU and 768GB memory). Each MM has 20 task slots. Previously we had the following configs on the peon: • druid.indexer.fork.property.druid.processing.buffer.sizeBytes=1000000000 • druid.indexer.fork.property.druid.processing.numMergeBuffers=12 • druid.indexer.fork.property.druid.processing.numThreads=12 We had an increase in query (~100 qps) but same amount of ingestion data coming in. We saw that we have a lot of query timing out (>30s), and also increasing kafka lag. We made the following changes: • -XX:ActiveProcessorCount=5 on the peon following this recommendation https://github.com/apache/druid/pull/12592 (ceil(96/20)=5) • druid.indexer.fork.property.druid.processing.numMergeBuffers=4 and druid.indexer.fork.property.druid.processing.numThreads=4 (leaving 1 for ingestion thread and a persist thread) • Changing maxRowsInMemory to 150,000 following this https://github.com/apache/druid/pull/13939 (previously we mostly hit 5 min persist threshold persisting about 200-450k rows each) • No change in number of tasks After the change, we see significant improvement in query response time and also no kafka lag However, we currently see that our broker throughput is less than router. Router has ~100qps while broker is at ~80qps. We tried doubling the number of broker 4->8 and increasing

druid.processing.numThreads=95

but doesn’t seems to help. Also tried increasing numMergeBuffers and numThreads on the peon, which also doesn’t seem to help. Any other idea? Thanks!

Each historical and each peon needs enough http threads so that they can accommodate all broker connections plus a bit more for inter-process communication.

good to hear the

maxRowsInMemory

change was helpful! and also good to hear that applying new defaults generally has been helpful 🙂 it tell us the defaults are improving.

good to hear the

maxRowsInMemory

change was helpful! and also good to hear that applying new defaults generally has been helpful 🙂 it tell us the defaults are improving.

They definitely are. That’s the problem with running old version. You only look for and find the better configs after the problem occurred. 😅

ABC Always Be Current 🙂