Hi questions on strange load accross VMs. Does anyone have faced this kind of issues before ? We’ve got 4 servers 2 of them shows lot of CPU/ RAM usages where the two others looks like normal/chill. Data is properly balanced (32 tables / 2 partitions (non spooled) with replication of 2 ) so each server is used correctly when queried. Can be seen on the network traffic close to equal on each server. Tried to rebalance tables but all already balanced. I’ve rework the JMX to grab only relevant data and I do not see anything. Same queries rate and segment processed / server. Any clues ?

Is the data skewed? If not, check if you have hot partitions. Or else take jfr and compare

no skew check both partition of each topic number of row quite similar

Group by $hostName to see if results are skewed

no skew tried and only a 1K ou maybe two diffrences betwen hosts. For non upsert tables got 4 server responses but for upsert table only two (changing randomly of pair).

Check entries scanned, ingestion rate, etc per server

ingestion is ✅ as kafka partitions are mapped to pinot partitions not skewed

Are you checking the metric graphs

For tables and $hostName running queries on pinot UI. For kafka partitiong skew using JMX of kafka.

No, there is a ton of Pinot metrics. CPU is used to ingest or query. Using Pinot metrics per server check what correlates with cpu

I know I’m browsing on them 😉 With JMX ans prometheus. But haven’t found something relevant 😇

Hmm, but that has to be because it is doing more / different work. The metrics like entries scanned should reflect that