the amount of extra space used should be about the same order of magnitude as the amount of data being ingest

note that for native batch (

index_parallel

or

compact

) you can reduce this usage by setting

druid.processing.intermediaryData.storage.type = deepstore

in MSQ (SQL-based ingest) we are adding something similar soon [we already have half of it implemented: using deep storage for stage outputs. but we're adding deep storage for sorting too and then we'll fully release the feature]