This message was deleted.

Too many small segments could negatively affect: • Query performance, lots of segment files to open/scan • Storage size, loss of economies of scale on column encoding dictionaries • Coordinator and Broker memory heap, as they hold a segment metadata cache • Metadata DB and interaction with it, if the number of segments is very large Are you running compaction to defragment your time chunks? How big are these segments? How many segments per time chunk are we talking about?

Use automatic compaction it will will reduce file size based on your granularity

These are post compaction segments. Our proof of concept system post compaction has around 5-10,000 segments per timechunk (30 minutes). Each segment is between 500MB > 1GB. We're still scaling things and have already reduced the granularity from an hour to 30 minute but I think to stay below the 2,000 segment per timechunk theory, we need to move to 1 or 5 minute granularity.

Hi @Nick M, Ideally each segment should have around 5 million rows as described here: https://druid.apache.org/docs/latest/operations/segment-optimization.html. How many rows are you ingesting every hour? Reducing the segment granularity will actually increase the number of segments. Think about it this way, if you have 5 million rows getting ingested every hour and your segment granularity is 1 hour, then you will have 24 segments for the day. If you reduce the segment granularity to 30 minutes then you will have double the number of segments in general. Also please check your kafka partition for data skews. Uneven partitions are very common when there is a data skew in kafka partitioning. I hope this helps.

We have 5 million rows per segment already. Our Kafka topic has around 1000 partitions (which is necessary to scale the ingest rate we need and probably also the root cause of the issue). In terms of rows per hour, we’re talking tens of billions. It’s a big system we’re developing for a large ISP…

Nick, In your case it sounds like you are already filling up the existing time chunks with large numbers of full sized segments. So ... making granularity finer in this case would help with query pruning ONLY if your queries are filtering on time ranges smaller than 30 minutes ... e.g if your queries are mostly looking at the most recent 5-10 minutes of activity, then pruning would work better if your segment was at the 5-10 minute granularity or smaller. If your __time value is being truncated to the same level as the segment granularity, and you are using range partitioning, then you may benefit slightly from smaller segments due to the records within the segment logically being sorted by the partitionig columns. The records are always logically sorted first by the __time value, so for this to happen all of the __time values within the segment should be the same value. You also want to make sure you don't lose the benefits of paralellism on the Historicals. For a given query, you want each Historical cpu to process N segments. If there are fewer segments than historical cpus then you are leaving resources unused to process the query. So, let's say you have 100 Historical cpus on your cluster, then for any given query, processing 500 or more segments should be good.

Hi @Nick M having around 1000 kafka partitions will definitely increase the number of segments that are getting generated during ingestion time. How many events are you ingesting per second? As can be seen in this blog post https://imply.io/blog/exactly-once-streaming-ingestion/ each kafka partiton should be able to handle roughly around 100K events. I have worked with systems that ingest millions of events per second with less than 100 kafka partitions. This presentation shows the design decisions that should be followed for those circumstances .

At the scale you're talking, where each segmentGranular bucket would be fully populated no matter what, the main downside to the smaller granularities is that it's more likely you'll get fragmentation due to late data or task rollover The way to think about it: • If late data occurs, you get a new segment created for each time chunk that sees late data. If you typically get late data over, lets say, 90 minutes, then segmentGranularity of "hour" leads to two segments being open. But segmentGranularity of "minute" leads to at least 90 being open • If tasks roll over crossing a segmentGranularity period, then the old and new tasks may both write some shorty segments. This is more likely to happen if segmentGranularity is short

We do have to contend with late data. We process the data before it hits kafka to ensure similar data ends up on the same partition which helps mitigate the scenario of fragmented data but not completely. This is where we're focusing a lot of our effort at the moment, in trying to ensure during ingest that segments are created as efficiently as possible with the minimum number of segments.

Did we ever track down what was causing the slowdown you saw with lots of segments in a single time chunk?

No - we couldn't get the coordinator to start successfully in the end (it kept running out of memory during startup probably due to the number of segments per time chunk). We trashed the cluster and started again with 30 minute granularity (and I feel we're in the same scenario again in that if I bounce a coordinator, I don't feel it'll come back as the 30 minute granularity is still seeing upwards of 20,000 segments per timechunk).

aargh. Let me see if we can repro something like this on our end. Although 20k is still a lot, so lowering your granularity further may be a good idea. Gut check— what's the average row size of those 20,000 segments? Like, are they very small or are they "reasonable" size (a few million rows)?

I think it's about half and half. 50% percent are a few million rows or more, and then the rest is made up of late data / fragmentation which causes the rest.