@Gian Merlino Any insight into this (you've been ridiculously helpful with my understanding of druid ingest to date)?

but, as far as i can tell, overriding it should work

i only read the code, i stopped short of actually trying it, so could be wrong 🙂

it looks like it's all wired up though for a default of 20M and for being overridable through

tuningConfig

so before fixing the docs i'd like to figure out what's going on with your situation

these are suppose to be the triggers for publishing: 1) any task has read more than

maxRowsPerSegment

(default: 5M) into a single segment id. [note this is doing a sum across all intermediate persists with the same segment id, not just looking at the in-memory data] 2) any task has read more than

maxTotalRows

(default: 20M, apparently) across all segment ids. [note this is per-task] 3)

intermediateHandoffPeriod

time has passed 4)

taskDuration

time has passed

what makes it look like (1) is not happening? i would guess that by the time you read 20M rows in a single task, you probably also read at least one segment with 5M rows. only way this wouldn't be true is if you're generating >4 segments, which isn't common (you'd need late data spread over a variety of time chunks in such a way that none exceeded 5M).