Hello, We’re running Apache Pinot 1.3.0 in produc...
# troubleshooting
p
Hello, We’re running Apache Pinot 1.3.0 in production. Cluster size: 2 brokers, 10 servers, 2 controllers, total docs ≈ 500 billion. Normal query performance is good, even at high volumes. However, we’ve started seeing a recurring issue: • A particular query on one table scans ~*40M docs* (numDocsScanned in broker metrics). • As soon as this happens, all queries start returning 504 Bad Gateway from brokers. • No errors in server/broker/controller logs — heap usage, GC logs look normal. • Restarting the Pinot servers immediately restores normal behavior. • The issue repeats whenever that same large-scan query is run. What we’ve tried so far: • Checked GC logs — no long pauses. • Thread dump during issue — query executor threads all busy with the large scan. • Added
pinot.broker.enable.query.cancellation=true
and slow-query logging. • Monitored broker/server metrics — no memory spikes, CPU high during issue. • Total partitions for this table: 10, total servers: 10. Questions: 1. Has anyone seen Pinot servers becoming unresponsive purely from a large scan, even when heap/GC are fine? 2. Is there a recommended way to limit docs scanned per query or abort queries early when scan is excessive? 3. Any known fixes/tunings in versions >1.3.0 for preventing one heavy query from starving all server threads? 4. Would increasing server query threads or adjusting segment assignment help, or just postpone the problem? Any insights, patterns, or configs we should look at would be appreciated.
m
It is probably not the scan but you might have an expensive query - group by with high cardinality, distinct etc. can you share the query?
p
since there are hundreds of queries, how do we figure out which query is causing that
m
Replied in the other thread
p
Hey, did you get answers to your questions @Pratheek Shetty