This message was deleted.

It's not single-threaded. I'm not sure of the details, whether it would always choose 7 (sub)tasks or not.

You can control the number of tasks in a native batch ingestion spec `index_parallel` with property

maxNumConcurrentSubTasks

. If you are using SQL Based Ingestion you can set the parallelism in the query context with property

maxNumTasks

. The input phase does not split up files, so for your case if you want 7 tasks to each process a days worth, you would need to split your weeks worth of data into at least 7 files.

Thanks @Ben Krug @Sergio Ferragut for the quick response. @Sergio Ferragut - I am interested in index tasks for now. so when we have 1 file with 7 days data index task will spin one java process with one thread and when we have 7 files with 7 days data index task will spin one java process with 7 threads or 1 thread(this is my doubt) Is this understanding correct? please validate

Each task (Peon JVM) sequentially reads the files assigned to it, you will need to increase parallelism (more tasks) in order to process multiple files in parallel.

To clarify, if you increase the number of subtasks, there'll be one main task (JVM), and the overlord will split up subtasks among different nodes and slots, and each gets its own JVM on whichever machine it's on. That's my understanding, I encourage you to check. (And let me know if I'm wrong!) For a quick mention of this idea, see here.