Hi Druid Team, I'm currently facing a workload were a kafka indexing task runs into OOM while allocating direct memory for segment decompression buffers. Here some context on why this happens: • While publishing, it tries to merge many small (<1000 rows) intermediate persists with 1000+ dimensions for around 1 Mio rows in total. • These small intermediate segments are created during intermediate persists were 99+% of the rows belong to large segments in the past (backfill) while the remaining (live) records only make up for 500-1000 rows per intermediate persist. • All of the live rows will end up in the same segment for publishing. • I'm aware that this backfill may better be performed as a batch ingestion - but in my application it will be quite common that large chunks of data containing old data will be added to the kafka topic without warning. Long story short: This problem could be easily solved by using the hierarchical segment merging algorithm that is used for batch ingestions and compaction tasks (by configuring

maxColumnsToMerge != -1

). Unfortunately, this setting is not available in the kafka ingestion spec. Are there any plans to make the hierarchical segment merging also available for kafka ingestion / stream ingestion in general?

If your segment sizes are already small and that's not an issue, have you considered setting maxRowsPerSegment and maxTotalRows to lower values such as 100k or less?

Hi thanks for the prompt response. Yes, I'm currently working around the issue by using maxRows/maxTotal... to limit the max. number of intermediate segments before a handoff. But I consider this just a workaround because the usual segment size will be around 2M rows for 700MB segment size, so decreasing the row sizes will create many segments in the "normal" scenario (this of course can be dealt with through compation which supports the hierarchical merging). This issue seems to appear only during low load edge-cases (for live data) when combined with the integration of large historical segments. But until the hierarchical merger is available, I will opt for smaller segments in order to guarantee system stability. -> This brings me back to my original question: Are there any plans to make this merging behaviour available for kafka indexing or should I plan to use the workaround for the forseeable future?

I'm not aware of such plans. This might be a good question for #C030CMF6B70 though

Ok thanks - I will repeat the question there. 👍

I've noticed a similar issue in the past but not related to any backfill data. This issue occurs for me any time I have an extremely large number of dimensions. I've always had to reduce the number of rows below the 5M mark. It's rare I have 100+ dimensions but when I do I always have to put a tighter limit on # of rows or risk an OOM.