This message was deleted.

75 segments with total size of 40MB seems like a lot of segments! generally that amount of data would be in one segment, which would be more efficient for this high concurrency workload

ok, thanks for the reply 🙂 I have some follow up questions, sorry • It sounds weird that 75 small segments are so slow. I read here that historicals would store the segments in memory. Do I need to configure something for that? • I ran the simplest compaction example job, and it was - waiting until I stopped the supervisor. I suppose it needs to lock the segments I want to compact. Is it possible to keep the compaction running in parallel to the supervisor? just making sure because I tried resuming the supervisor while the compaction task was running and it eventually failed. • What is the difference between reindex job and compaction job. I assume that reindex can also create new partitions for the segments. After running the compaction, it's 2 segments now, and testing it a bit, I see that the performance did improve when the cluster was huge, but it was still very slow. When we ran around 100 queries some queries took 15 seconds still. I tried to shrink the cluster to 2 historicals, 2 brokers, with historical config

jvmMaxDirectMemorySize: 3G
  jvmMemory: '1G'
  runtime:
    bufferSizeBytes: 100MiB
    processingThreads: 4
    httpThreads: 150
    cacheSizeBytes: 256MiB

and broker config:

jvmMaxDirectMemorySize: 2G
  jvmMemory: '2G'
  runtime:
    bufferSizeBytes: 100MiB
    numConnections: 60
    processingThreads: 2
    httpThreads: 70

and I see that the scan times spike to 1.5 seconds, the scan pending is very low now, and the CPU usage is also low. The

query/cpu/time

is quite high, 160ms. Not sure whether it's bad. What more can I look at to understand the bottleneck?

some thoughts:

It sounds weird that 75 small segments are so slow

this is mainly due to per-segment overheads: each segment has its own dictionary, indexes, etc, & it's generally faster to amortize this stuff across multiple segments

I suppose it needs to lock the segments I want to compact. Is it possible to keep the compaction running in parallel to the supervisor?

compaction and active ingestion do indeed mutually lock out: so, generally, when using auto-compaction people will use the

skipOffsetFromLatest

config such that they don't run for the same time ranges when doing a manual compaction just for testing purposes, generally I suspend the supervisor while compacting to keep things simple (& then use

skipOffsetFromLatest

in production)

What more can I look at to understand the bottleneck?

to analyze bottlenecks i like to use flame graphs: https://support.imply.io/hc/en-us/articles/360033747953-Profiling-Druid-queries-using-flame-graphs if you capture these on your Broker and Historical, feel free to post them here with your query and we can see if we have additional suggestions

One more thing, we are doing some sub-querying to group by some columns.

this may be relevant, if your subqueries are returning larger amounts of results, that can create a bottleneck (as all subquery results need to be written into merge buffers and then sent out to the Broker) if you're in this situation, you could potentially benefit from more merge buffers, or from new option in 24.0 (not out yet! but soon 🙂) added here: https://github.com/apache/druid/pull/12513 (

mergeThreadLocal

). each of these things targets different bottlenecks, so the thing that would benefit you the most could be learned from the query + flame graphs

That indeed help a lot. I will test these tnigs out

There are several layers of onion to peel. 1. the latest build has Jetty metrics. See if you still have plenty of Jetty threads. 2. There are a lot of metrics that said xyz pending time. That will help you see at which stage you are stuck on. 3. If this is on AWS, see the EBS timing metrics, who knows you are stuck when loading data from segment-cache.