This message was deleted.

increase druid.indexer.fork.property.druid.processing.buffer.sizeBytes (max is 2 GB). If you increase this to 2GB and still see this error then reduce max rows in memory and also consider indexing only fields used in queries

Thanks! Trying now. Out of curiosity, what's the reasoning behind that?

I don't think that increasing

druid.indexer.fork.property.druid.processing.buffer.sizeBytes

will help; if you are out of direct memory then increasing buffer size will only increase the demands on buffer memory. Looking at your settings, your current buffer size (134217728) and count (1) is not too big relative to your direct memory size (4g). So I would guess the direct memory is being used by column merging. You can read more about that here: https://apachedruidworkspace.slack.com/archives/C0309C9L90D/p1663951074038569?thread_ts=1663936671.637089&cid=C0309C9L90D TLDR: if that's the issue, you can mitigate it in two ways: 1) Set

maxColumnsToMerge

using the guidelines from that thread I linked. 2) If this is batch ingest: Use MSQ / SQL based ingest instead of native batch ingest. It automatically tunes column merging to avoid running out of memory, so you don't need to worry about it

It is a batch ingest. I would use MSQ, but I'm using a custom agg function.

Good to hear! BTW, you can use custom agg functions in MSQ, albeit with a bit more work. You would need to define a SQL binding for your agg function

Yes, we're working on that 🙂