Hi! I have a question about parallelism in MSQ ingestion. My general understanding is that there are 2 levels of parallelism: number of tasks for given query (maxNumTasks) and number of processing threads in each task (maxThreads). There are some details I am not sure about though.

When reading external data, EXTERN can read multiple files in parallel across different worker tasks. However, EXTERN does not split individual files across multiple worker tasks

- docs - We should have number of tasks between 2 (1 controller + 1 worker, minimal parallelism) and number of files we are reading (maximum parallelism). - Question 1: If we have 1 task with 4 threads can it read 4 files at once? - Question 2: If we are reading 10 files and writing 20 segments - can segment writing also take advantage of parallelism? Shouldwe have 10 tasks each with 2 threads then?

Worker tasks run single-threaded, which also determines the maximum number of processors on the server that can contribute towards multi-stage queries.

- docs - Question 3: Does this relate to default of maxThreads = 1? Or does it mean something else? - Question 4: I am confused. There 3 different mentions of a default thread count for a worker and they give slight different answers. Here we have mention of "single-threaded", in code there a default of 1 thread, here we have "Not set (use default thread count)" (does it mean 1 or

druid.processing.numThreads

value?)

Indexing service tasks of type

query_worker

that execute a query. There can be multiple worker tasks per query. Internally, the tasks process items in parallel using their processing pools (up to

druid.processing.numThreads

of execution parallelism within a worker task).

- docs - We can have more than 1 thread per task if we want to parallelize some stage of a query. This can be set in query context (capped at

druid.processing.numThreads

config value set for Peon that picks up the task). - Question 5: What is the benefit of higher

druid.processing.numThreads

? What query stages can be parallelized? If I am doing simple ingestion (few files, simple select with no transformations) would I benefit from higher thread count at all?