This message was deleted.

Hi @Olivier JOUSSET, I suggest you refer to https://druid.apache.org/docs/latest/operations/basic-cluster-tuning.html •

nb core

refers to number of CPU cores per node types • This is a great reference for managing your worker capacity : https://www.druidforum.org/t/druid-worker-capacity-configuration/8000

Thanks for your anwser. I'm a little bit confused because this rule is valid assuming that Druid runs alone on the node... We've 112 core on the node but there are many ohters pods running... so how can you know the right value

I’m a little bit confused because this rule is valid assuming that Druid runs alone on the node

not really, Druid runs on multiple pods as well.

so how can you know the right value

It depends on what you are tuning : query, ingestions etc ?

Issue we have right now is on

druid.router.http.numMaxThreads

. We faced a Insiffucient configured threads error at router startup. It was set to 50. According to the formula

max(10, ((number of cores * 17) / 16 + 2) + 30)

we increased it to 151 (as we have 112 core on the node) and it works... but I'm surprise that the number of core of the node is consider here.

That can result to CPU contention for other pods running on the same pod no ?

Yes, that can happen

how many cpus have you assigned to the container running the router?

You mean as Request/Limits ?

yes

the limit is 2

I would suggest leaving httpthreads at default. Http threads are not active continuously and especially on the router do hardly any work. I would focus on the processing threads in the historicals….these are the constantly working threads that matter a lot for performance. Also understand your concurrency and query sla expectations and use that to determine the number of queries the historicals need to process at any one time and use that to set http connections on the broker and http threads on the historicals.

ok so regarding the

druid.router.http.numMaxThreads

it is not a problem to let the 151 as they will never be used ? We'll look at your recommendation for historicals part

yes…http threads at 151 should not be a problem. The thread will simply pick up a query from the jetty queue and then pass it to the broker and become dormant. Then when the broker is done the thread picks up the results and passes it on to the clients. Http threads on historicals also to merging of the intermediate agg from the processing threads. So focus on the historicals.

I would recommend just upgrading to Druid 25 and using the auto tune script

Thanks @FJ. Any documention on that. Didn't find anything on Druid 25 release note

you can use the start-druid script in bin the usage is documented in the script

sample usage:
    start-druid
            Start up all the services (including zk).
            services config is read from conf/druid/auto.
            zk config is always read from conf/zk.
    start-druid -m=100g
            Start up all the services (including zk)
            using a total memory of 100GB.
    start-druid -m=100g --compute
            Compute memory distribution and validate arguments.
    start-druid -m=100g -s=broker,router
            Starts a broker and a router, using a total memory of 100GB.
    start-druid -m=100g --s=broker,router \\
    -c=conf/druid/single-server/custom
            Starts a broker and a router, using a total memory of 100GB.
            Reads configs for each service (jvm.config, runtime.properties)
            from respective folders inside the given root config path.
    start-druid -s=broker,router \\
    -c=conf/druid/single-server/custom
            Starts a broker and a router service, reading service configs
            from the given root directory. Calculates memory requirements for
            each service, if required, using upto 80% of the total system memory.
    start-druid -m=100g \\
    -s=broker,router,zookeeper \\
    -c=conf/druid/single-server/custom \\
            Starts broker, router and zookeeper.
            Configs for broker and router are read from the specified root directory.
            Config for zookeeper is read from conf/zk.

Seconding this recommendation. The new automatic start script is way simpler than tuning everything yourself!