This message was deleted.
# generals
Slackbot
09/20/2023, 11:44 AMThis message was deleted.
j
jagr
09/20/2023, 11:50 AMI'm also curious about
Copy code
For good performance, you will want enough Historicals such that each Historical has a good (free system memory / total size of all druid.segmentCache.locations) ratios
Sergio Ferragut
09/20/2023, 11:00 PMFrom a performance perspective, the higher the better. A ratio of 1 means you can cache all segments in memory.
Sergio Ferragut
09/20/2023, 11:09 PMSizing estimates are hard, there are too many assumptions that need to be made which are usually wrong. A sizing strategy that I subscribe to:
• Define SLAs
• Create a small cluster
• Test with your data at a 5-10% of production volume
• Tune the data, tune the queries
• Measure performance with expected concurrency varying query parameters to simulate varying user activity (JMeter is good for this)
• Scale components of cluster if needed
• Double the data
• Measure again and scale again to achieve SLAs
You now have the scalability of your workload at two data volumes and you can extrapolate to the production level.
a
Arun C
09/22/2023, 12:14 AMId agree with everything sergio said, I normally go with a 1:10 ratio (for every 1 gb of memory 10gb of disk ssd or nvme) and this has served us pretty well for running subsecond queries against druid on relatively large datasets (700B-1T+ range). But id really stress the 4th bullet point in his recommendation for determining an optimal node configuration size
• Tune the data, tune the queries - This really means ensuring you compact/partition your data appropriately based on data access patterns which surprisingly many new druid operators tend to miss when benchmarking the resource utilization of their cluster. This has (From my experience), often led to Druid operators overprovisioning resources for their cluster and wasting resources either due to having too many segments or the partitioning of the segment just being configured sub optimally. https://imply.io/blog/multi-dimensional-range-partitioning/ is a really good blog post that helps you understand the differences between partitioning strategies in druid.
Druid by default if you are benchmarking batch or hadoop based indexing jobs will normally perform a hashed based partition strategy based on all dimensions in your dataset. I normally find this configuration to not be optimal for the vast majority of use cases due to low data locality especially when Druid operators are attempting to achieve a p95 >1s query results on massive datasets. Understanding this is pretty critical for benchmarking the resource requirements of your cluster.
For real time ingest it'll default to dynamic partitioning which once again may not be performant for your needs without running a hashed/ranged based auto compaction job to further optimize the segment. Once again this can come at a pretty large cost due to no data locality being enforced.
s
Sergio Ferragut
09/22/2023, 12:17 AM👆 🙌
j
jagr
09/22/2023, 8:32 AMthanks a lot 🙂 that's true, biggest performance improvement that we got was from compaction tasks
a
Arun C
09/22/2023, 3:02 PMAnother big brain trick is to sort order the dimensions based on cardinality of the column to get better compressible runs and potentially reduce cpu cycles when filtering on those dimensions
Arun C
09/22/2023, 3:05 PMThis is because druid sorts rows within each segment based on the dimension order you provide. So if there is a natural column u use for filters or group bys make it first