Hi, We have setup our Apache Pinot Cluster with 1.0.0 version. The query info - We are having 1 fact table, 2 dimension table for joining We are currently performing the load testing on our query With the concurrent users where each request is returning 0.1 M records We tried first with 5 concurrent users all test passed But later with 10, 20, 30, ..., 100 We see 50%-80% test cases were failing with 5XX error. When we checked the issue it was because of the memory consumption was very high by the server. Instead of increasing the memory and trying to see the ideal case for performing the load test. Is there any better way we can work with the concurrent users tests Sharing Pinot Server JVM Config -

jvmOpts: "-Xms5120M -Xmx8192M -XX:+UseG1GC -XX:MaxGCPauseMillis=200 -Xlog:gc*:file=/opt/pinot/gc-pinot-server.log"

cc: @Ashwin Agalcha @Shubham Setia @Sambasiva Mukkadala

@Xiang Fu ^

in general it means that the queries are exhausting cpu or memory. One thing you can try is to set a table level query quota Ref: https://docs.pinot.apache.org/configuration-reference/table#Quota https://docs.pinot.apache.org/operators/operating-pinot/oom-protection-using-automatic-query-killing

@Xiang Fu But we don't want to actually kill such queries. The requirement is to run these queries with record limit of 100000 and high concurrency of 20+ users.

Correct, the purpose here is to increase latency instead of query failure. I think you can test for the qps with 1 replicas, then if you want to increase the qps, you need to add more replicas.

We have performed the load testing and found out the below results. Is there any way we can improvise the pinot setup to have successful results for the concurrent users. @Xiang Fu

If you can tolerate, you can set the timeout to be longer for each query. Pinot default sets the timeout to 60 seconds

Is there any other way apart from the query timeout to handle the concurrent users load in pinot Like some setting in apache Pinot where it is not able to serve too many concurrent users requests One strange observations we have seen is that even after the memory consumption is very less one of the server is becoming dead while serving the request @Xiang Fu cc: @Ashwin Agalcha @Shubham Setia

you can set query qps quota for table, which will slow down your query

@Xiang Fu We are having the api layer written in python that is querying the pinot using swagger api. We have found the below results in terms of response time and response file size. We wanted to reduce the response time and file size. The query consists of 1 fact table which is performing lookup with 2 other dimension table. Is there a way to return the response in compressed form to reduce the file size cc: @Sambasiva Mukkadala @Ashwin Agalcha

This is good suggestion! Right now this is not supported yet, can you create a github issue and we can pick it up upon the resource availability