Hi team, we’re seeing sudden CPU spikes (~150%) on Pinot brokers without any change in QPS or query patterns, leading to broker restarts and production downtime. Looking for insights on possible causes and preventive measures.

Sudden cpu spike typically points to some on-off expensive query, check for server logs to see if there are any traces. Also check

pinot_broker_totalServerResponseSize_OneMinuteRate

,

pinot_server_numEntriesScannedPostFilter_OneMinuteRate

and

pinot_server_numEntriesScannedInFilter_OneMinuteRate

metrics to see if there is any correlation with the spike

Yes, during debugging I found a dashboard query that calculates unique user traffic based on the provided time range and time grain. It was using

COUNT(DISTINCT ...)

, which is a very heavy operation. I’ve changed it to use HyperLogLog (HLL) instead.