https://pinot.apache.org/ logo
Join Slack
Powered by
Hi Pinot Team , We are migrating from Elasticsearc...
# troubleshooting
r
Hi Pinot Team , We are migrating from Elasticsearch to Pinot . We use cursor pagination (POST /query/sql?getCursor=true) on V1 to fetch large result sets including the requestId column (primary key). We originally used V2 (/query) for cursor, but V2's multi-stage engine fails when requestId is in the SELECT clause — the query is cancelled during plan setup (EMPTY_MAILBOX_SEND, error 503, "CANCELLED: client cancelled") because requestId conflicts with V2's internal gRPC stage-tracking identifier. Alias, CAST, and CONCAT wrapping all failed. Per official docs, cursor is a V1 feature, so we switched to /query/sql?getCursor=true. The initial cursor query works correctly and returns the first page. However, subsequent pagination GETs to /responseStore/{requestId}/results fail intermittently with "Query results for {requestId} not found". Out of 12 concurrent Lambda invocations (1 hour split into 5-min windows), some succeed fully, some fail immediately, and some partially succeed (e.g., 600K of 1.4M records fetched before failure). @Mayank @Xiang Fu
j
Hey, Also, do you see intermittent failures or all failures start after a fixed time after cursor creation? Additionally, do you have broker affinity in your lambda request? Cursors are broker local and the subsequent requests should be routed to the same broker itself.
r
"Cursors are broker local and the subsequent requests should be routed to the same broker itself" yes exactly what is happening. Since we are behind nlb which does not have sticky routing it is not going that particular broker .
@Jinesh Parakh 1. Now with our case is it possible to ensure pagination requests route to the same broker ? 2. Is requestId as a column name a known conflict with V2's cursor/multi-stage engine? although I read single stage is for cursor based paginated query but I have tested fetching of multiple records using multi-stage engine but when added requestId in the select query , started getting this error pasting the log
Copy code
[PINOT-DEBUG] raw response: {"numRowsResultSet":0,"partialResult":true,"exceptions":[{"message":"Received 1 error: Cancelled by sender with exception: CANCELLED: client cancelled","errorCode":503}],"numGroupsLimitReached":false,"numGroupsWarningLimitReached":false,"maxRowsInJoinReached":false,"maxRowsInWindowReached":false,"timeUsedMs":1946,"stageStats":{"type":"MAILBOX_RECEIVE","parallelism":1,"clockTimeMs":0,"children":[{"type":"MAILBOX_SEND","stage":1,"parallelism":1,"fanOut":1,"rawMessages":2,"serializedBytes":22708835,"serializationTimeMs":109,"clockTimeMs":0,"children":[{"type":"SORT_OR_LIMIT","executionTimeMs":486,"emittedRows":293645,"parallelism":1,"clockTimeMs":486,"children":[{"type":"MAILBOX_RECEIVE","executionTimeMs":447,"emittedRows":293645,"fanIn":90,"inMemoryMessages":1,"rawMessages":122,"deserializedBytes":44869955,"deserializationTimeMs":36,"upstreamWaitMs":8414,"parallelism":1,"clockTimeMs":447,"children":[{"type":"EMPTY_MAILBOX_SEND","stage":2,"description":"No stats available for this stage","table":"URDMCore"}]}]}]}]},"maxRowsInOperator":293645,"requestId":"1832875796000000000","clientRequestId":null,"brokerId":"Broker_pinot-broker-3.pinot-broker-headless.reolaps-pinot-cluster-1.svc.cluster.local_8099","numDocsScanned":0,"totalDocs":0,"numEntriesScannedInFilter":0,"numEntriesScannedPostFilter":0,"numServersQueried":90,"numServersResponded":90,"numSegmentsQueried":0,"numSegmentsProcessed":0,"numSegmentsMatched":0,"numConsumingSegmentsQueried":0,"numConsumingSegmentsProcessed":0,"numConsumingSegmentsMatched":0,"minConsumingFreshnessTimeMs":0,"numSegmentsPrunedByBroker":0,"numSegmentsPrunedByServer":0,"numSegmentsPrunedInvalid":0,"numSegmentsPrunedByLimit":0,"numSegmentsPrunedByValue":0,"brokerReduceTimeMs":589,"offlineThreadCpuTimeNs":0,"realtimeThreadCpuTimeNs":0,"offlineSystemActivitiesCpuTimeNs":0,"realtimeSystemActivitiesCpuTimeNs":0,"offlineResponseSerializationCpuTimeNs":0,"realtimeResponseSerializationCpuTimeNs":0,"offlineTotalCpuTimeNs":0,"realtimeTotalCpuTimeNs":0,"explainPlanNumEmptyFilterSegments":0,"explainPlanNumMatchAllFilterSegments":0,"traceInfo":{},"tablesQueried":["URDMCore"],"offlineThreadMemAllocatedBytes":0,"realtimeThreadMemAllocatedBytes":0,"offlineResponseSerMemAllocatedBytes":0,"realtimeResponseSerMemAllocatedBytes":0,"offlineTotalMemAllocatedBytes":0,"realtimeTotalMemAllocatedBytes":0,"pools":[],"rlsFiltersApplied":false,"groupsTrimmed":false}
j
1. Now with our case is it possible to ensure pagination requests route to the same broker ?
Cursor responses already return broker host and port. Can you bypass the load balancer and route requests to specific brokers?
Or if shared storage latencies are acceptable, you can try moving the pagination dir to s3 or some other shared storage.
r
The brokerId in our response is 
Broker_pinot-broker-1.pinot-broker-headless.reolaps-pinot-cluster-1.svc.cluster.local_8099
. Our Lambda accesses Pinot through a VPC endpoint │ (NLB). Can Lambda resolve this K8s hostname? Or is there an IP/external address we should use for direct broker access?
j
I'm not an expert in lambdas. iirc the response should have 
brokerHost
and 
brokerPort
fields. Can you share that? What's the value for 
pinot.set.instance.id.to.hostname
?
Is using shared storage not feasible?
r
Actually latency is a concern in our case.
🆗 1
can you confirm this - Is requestId as a column name a known conflict with V2's cursor/multi-stage engine?
"you can try moving the pagination dir to s3 or some other shared storage." - Can you elaborate more on this , not familiar with this approach.
j
can you confirm this -
Is requestId as a column name a known conflict with V2's cursor/multi-stage engine?
Will check tomorrow
"you can try moving the pagination dir to s3 or some other shared storage." - Can you elaborate more on this , not familiar with this approach.
I meant you can try storing the cursor responses on s3 kinda shared storage. With same s3 dirs for all brokers, any broker should be able to access the requests. iirc broker affinity is not checked during cursor read path. 
pinot.broker.cursor.response.store.file.data.dir
pinot.broker.cursor.response.store.file.temp.dir
and other necessary configs. Ref: https://docs.pinot.apache.org/reference/configuration-reference/broker
r
Thank you for your explanation , will read further.
2 Views
Open in Slack
PreviousNext
Powered by