Hi Team, In Segment size optimization documentation, the following is mentioned

- Number of rows per segment: it's generally recommended for each segment to have around 5 million rows. This setting is usually more important than the below "segment byte size". This is because Druid uses a single thread to process each segment, and thus this setting can directly control how many rows each thread processes, which in turn means how well the query execution is parallelized.

- Segment byte size: it's recommended to set 300 ~ 700MB. If this value doesn't match with the "number of rows per segment", please consider optimizing number of rows per segment rather than this value.

Some clarification questions around it - • Segment here is one partition? In the screenshot I shared below, we have 5 partitions for a

day

segment granularity. Is each segment partition (we have 5 in the screenshot) one segment? • Is Segment byte size mentioned in the documentation for one replica? We have two replicas, so the segment sizes of one replica for each partition in the screenshot is between 300 to 700 MB. Is that right? (the reason why we did not consider 5 million rows in this case is because it was creating only one partition reducing filtering when range partitioning is applied).

segment is one partition. basically one row in your screenshot. So you have 5 segments per DAY

Hi Mounika, It also depends heavily on your queries ... Sometimes you want range partitioning if you query across a lot of time intervals and want to limit the number of segments that they query needs to open. In the case Range partitioning on the key that matches the query filter will help reduce the number of segments opened by the query Sometimes you want the query to open more segments to gain parallelism. In this case range partitioning may work against you, possibly better to use Dynamic partitioning to spread the data out more. Size of segment generally doesn't affect query filter/access time ... the dictionary indexes on dimension columns have binary lookup capability so increasing the number of rows and/or cardinality doesn't slow it down much, so long as the filter is selective. So in this case fewer, larger segments should be okay. If the filter is not selective and you are using a large percentage of the rows in each segment, then again more, smaller segments and dynamic partitioning will help provide more parallelism for sifting through the data, at least at the lowest level of query execution.

@Maytas Monsereenusorn The segments UI shows total size of all replicas. So in this case, one replica's segment size is 500MB (1GB for 2). Thank you for confirming my understanding! That was helpful.

Imo, "a lot of time intervals" is relative... 1 week - 1 month is pretty common. If you're mostly querying for 1 year, that might make a difference. Best to test different sizes, if possible and called for.

Got it. Will evaluate our queries accordingly. Thank you!

You're welcome! Feel free to post any results or questions!

> May I know if you can share any insights on what interval you think as a lot? We typically receive from 1 week to 1 month and a lot of times 6 months to 12 months (sometimes, 1.5 years as well). Just to elaborate on this a bit more with an example -- Supposed you have a lot of data, so 30 segments per day, and you are storing with with DAY level segment granularity. And your query runs over a 60 day time interval. Without range partitioning your query would be opening 60x30 = 1800 segments. But with range partitioning on your filter key you might only be opening 60x1 = 60 segments.

Got it. Thank you!