This message was deleted.

workers are new jvms that run indexing jobs.. --- I'll try to briefly respond to that although most correct answers exist for sure: • Druid indexing happens as a result of indexing tasks which pull data from various sources • In a typical Druid setup, MiddleManager (MM) is the module responsible to spawn and manage such tasks (each task runs in a new jvm process) • Thus the worker limit is applied by the MM So, there are several things to consider re: capacity: 1. Parallelism, how many jobs is your cluster capable of running (for a 3 quadcore cluster of MMs we can naively say that 12 workers may have the chance to run in parallel at most - so it makes an honest default) 2. If you have more than 12 indexing jobs then only 12 will run and the rest will wait on a "queue" till the previous complete (let's say this takes 1 hr) 3. And it goes on 4. You could increase this capacity to 24 and then all jobs would start, again the servers will only run 12 parallel processes at any given time and preemptively switch, but especially for streaming you'll have gains and you'll see data sooner in your results 5. But then you'll also have to check your memory constraints, one jvm for each task makes this workers pretty heavyweight so keeping them to sane limits may be better --- 1. there is (was?) an experiment with an Indexer process that was capable to run more than 1 tasks per jvm, so point no6 would be lightly considered 2. there were task isolation considerations tho and I'm not sure if we're going to see it happening 3. another approach is for environment with existing orchestration (see K8S) to run Druid clusters without MM and the host orchestrator run the processing tasks. You still have similar resources and constraints but managed and scaled from a single point 4. not sure about non-K8S MM-less setups. I want to find some time to experiment with a similar approach for Hashicorp's nomad --- the docs are explaining most of the above points (and way better than I do) with all the configuration details required to make it work as you want

thank you Aggelos for response.