Hi I wanted to increase parallelism (I am playing with

segmentCreationJobParallelism

and

pushJobSpec/pushParallelism

) in my batch ingestion, but even though my machine has 32 cores, speedup above parallelism of 2 is not significant. This log message gives me a hint that my configuration is not entirely correct

Creating an executor service with 2 threads (Job parallelism: 4, available cores: 2.)

How to tell the executor in the controller it has more cores available?

The log seems to suggest only 2 cores?

So, it means it is only able to detect 2 cores whereas you have 32 cores, intresting.

To be clear, Pinot will just call the Java api to get num cores. So if that api is returning 2, it is probably something outside of Pinot

Ooopsie our mistake 😅 we had

-XX:ActiveProcessorCount=2

in our setup