We have a Pinot / Kubernetes deployment with 6 con...
# troubleshootingt
Tony Requist
11/10/2021, 3:26 PMWe have a Pinot / Kubernetes deployment with 6 controller pods. We are seeing high CPU on one controller, very low on the others. Restarting pods does not change this behavior. Our Pinot is now primarily ingesting one fairly high volume Kafka stream with 128 partitions. Is this expected?
x
Xiang Fu
11/10/2021, 3:46 PM
It's expected I think. Pinot controller doesn't do actual ingestion work but handles the management of segment assignment, you should expect server side to ingest data.Also a lot of heavy lifting works are done by lead controller, which is elected automatically. Typically we don't do more than 3 controllers.
t
Tony Requist
11/10/2021, 3:47 PMThanks. I did find that this is the leader. I will switch to 3 controllers but make them larger.
x
Xiang Fu
11/10/2021, 4:11 PM
👍
s
Subbu Subramaniam
11/10/2021, 6:34 PMHow many tables do you have? If you have many tables, the load will be divided roughly equally amongst the controllers for each table. If you have only one table, then one of the controllers will be doing all the work, of course.
Subbu Subramaniam
11/10/2021, 6:36 PMAlso, think of the work in the controller as "meta" work. So, it is proportional to the frequency new segments added/deleted (roughly). And then there are periodic jobs that are sort of proportional to the number of segments you have in tables (data size does not matter). It is useful to check if your realtime tables are creating segments too frequently.
Subbu Subramaniam
11/10/2021, 6:36 PM@Tony Requist ^^
t
Tony Requist
11/10/2021, 6:50 PMCurrently we have essentially one table (2 tables but one is 98% of the volume) but that will be changing over time.
So as we add more tables we will see load more distributed over controllers.