Hi Pinot Team. We are running a performance test. The latency of slow queries (2s to 7s) is much higher than the average response time (62.5ms). The graph looks like a warm-up. When a large number of requests are sent, the latency of the initial requests of the test is high, and the subsequent requests show an average response time. Here are the two Pino-Server query logs. I have noticed that the pinot-server is causing delays.

Processed requestId=5111,table=table_poc_OFFLINE,segments(queried/processed/matched/consuming)=785/419/193/-1,...totalExecMS=4113...numDocsScanned=229,...,scanPostFilter=1832

Processed requestId=34700,table=table_poc_OFFLINE,segments(queried/processed/matched/consuming)=784/432/163/-1,...totalExecMS=141...numDocsScanned=177,...,scanPostFilter=1416

Is there any way to reduce this high latency difference? The table was created with a star-tree index(4 dimension columns, SUM__). 159 dimension columns, 47 metrics columns, 2 date-time columns 3.2T (3 replicas) 2,354 segments 834,221,210 docs 3 pinot-server, heap 16G 3 pinot-broker, heap 8G (the same hardware as pinot-server)

Singe you have only one server per replica and loading > 1 TB of data I feel you are IO bound

@Mayank Our cluster is on-premiss(not kubenetes) and uses ssd . Our problem is that the response time variance of requests is too large. Could it be that pino-server has this delay in mapping the target segment to mmap or loading it into memory to process the query request? (like segment warming up..)

Please share the query response metadata for a slow query. also what’s the cluster config (how many servers, their jvm config etc)

pinot-server hardware spec : 40 vcore, ssd, mem 256G pinot cluster : • 3 zookeeper, 3 controller ( the same hardware ) • 6 pinot server Test Tenant : • 3 pinot-server, heap 16G(-Xmx, -Xms) • 3 pinot-broker, heap 8G (-Xmx, -Xms, the same hardware as pinot-server) To get query response metadata, we have to re-test. I'll run the test again and give it to you. These two Pino server logs are the same query, only the condition values in the predicates are different. Could we find the cause here?

Processed requestId=5111,table=table_poc_OFFLINE,segments(queried/processed/matched/consuming)=785/419/193/-1,...totalExecMS=4113...numDocsScanned=229,...,scanPostFilter=1832

Processed requestId=34700,table=table_poc_OFFLINE,segments(queried/processed/matched/consuming)=784/432/163/-1,...totalExecMS=141...numDocsScanned=177,...,scanPostFilter=1416

---------------- We don't think of it as a characteristic of a table or data. The reason is that the response results of the same query with the same condition value of the same predicate in the test are highly different. ---------------- Attached image is another test result of sending request for 5 minutes and running 4 tests. Restarted the pinot-server before the second test (with a response time of more than 2 seconds).

Let’s discuss offline