Hi all, I’ve set up a Trino + Pinot cluster where ...
# troubleshooting
j
Hi all, I’ve set up a Trino + Pinot cluster where we query and join data from Pinot and other databases. But we’re hitting a bottleneck during large table scans from Pinot via gRPC. When Trino runs a query that needs a full table scan on Pinot, it first gets the list of all segments. Then, based on the total number of segments and the pinot.segments-per-split config, it sends multiple parallel gRPC requests to the Pinot servers. This is overwhelming the Pinot servers and causing them to choke. I can’t find any config either on the Trino or Pinot side to control how many parallel requests Trino sends to each Pinot server. I know increasing pinot.segments-per-split could reduce the number of request, but Pinot’s gRPC implementation loads all the segment data into Netty memory before streaming, and there’s a 2GB cap by default. So if I increase segments-per-split too much, Request fails due to netty issues. On the other hand, reducing segment size won’t help either — it just leads to even more segments, and that means more parallel requests from Trino, which again overloads Pinot. Is there any config or workaround to throttle the number of parallel Trino requests going to a Pinot server? Ideally, I’d want Trino to send only N requests at a time to each server and wait for them to complete before firing more or Pinot to queue above threshold request and server them by priority trino-pinot connector config
Copy code
pinot.connection-timeout=60s
pinot.aggregation-pushdown.enabled=true
pinot.non-aggregate-limit-for-broker-queries=2147483647
pinot.max-rows-for-broker-queries=2147483647
pinot.non-aggregate-limit-for-broker-queries=2147483647
pinot.grpc.port=8090
pinot server config:
Copy code
pinot.query.scheduler.name=fcfs
pinot.query.scheduler.query_queue_size=16
pinot.query.scheduler.query_runner_threads=2
pinot.query.scheduler.query_worker_threads=4
pinot.server.query.executor.max.execution.threads=2
pinot.server.grpc.max.inbound.concurrent.calls=8
pinot.server.grpc.numWorkerThreads=4
pinot.server.grpc.numBossThreads=1
none of the above config is working to control grpc request.