This message was deleted.

Try setting: maxNumConcurrentSubTasks I am assuming you have not touched the split hint spec https://druid.apache.org/docs/latest/ingestion/native-batch#split-hint-spec

the subtasks are spawned after the segments are pulled, I'm talking about what happens just before, when segment data is downloaded from object storage

Are you doing auto compaction ?

yes indeed

If yes then if you provide the schema then the first stage will not download the segments

hmm so you mean that it's downloading all the data to auto discover the structure of the data over the interval ?

Yes

That's a valid option, if we don't forget to update it when necessary 🙂 And otherwise no way to have this download happen in a parallel way ?

This is due for a revamp. Active work is being done here revamping the segment metadata so that it resides in the coordinator which launches the auto compaction. So may be we are able to finish that work by druid 29

ok sounds good enough