This message was deleted.

Hmm - maybe have a look at https://blog.hellmar-becker.de/2023/06/28/indexes-in-apache-druid/

Hi Amos, to add a bit more commentary to Hellmar's Blog ... Druid does not have secondary indexes as separate objects like relational databases do ... it stores data in columnar format and all dimension columns (the ones you filter on) are individually indexed by dictionary encoding for high speed lookup. All of the cross-referencing between "index" structures and the data within and across columns is all contained within the individual Druid segment. It's a very compact and efficient storage design.

Thanks! I guess Druid will rebuilt all those bitmaps during compaction from scratch then.

Yes, compaction merges dictionaries and rebuilds indexes for each dimension.

That could potentially be expensive because of write-amplification. What's the default max size of a segment in Druid?

The default target size is usually 500mb and/or 5m rows of data ... but I have seen segments stored at up to 2GB size. Compaction itself is not considered too critical a performance area of the product (e.g vs query performance) ... but it does need to keep up with the ingestion rates so there are tuning parameters that can apply. Compaction performance is also highly dependent on the data being compacted and the desired time granularity and data partitioning schema. So there is a lot to experiment with here if you would like to dive into this area of the product. Also, the new Multi Stage Query Engine (MSQ) can be used for compaction, and reports are that it runs much faster than native compaction jobs.