https://pinot.apache.org/ logo
Join Slack
Powered by
<@U0A4H6DCAGN> <@UDQU92KBK> <@UGRJA9TEH> Hi Pinot ...
# troubleshooting
r
@Jinesh Parakh @Mayank @Xiang Fu Hi Pinot team, We're running into a blocking issue with the V2 multi-stage engine and need guidance. Our Setup: • Pinot cluster with 18 brokers behind a Network Load Balancer (NLB) • Table: urdmCore with requestId as the primary key column • Query volume: ~1.4M+ records per query, using cursor-based pagination • Client: AWS Lambda (Java) The Problem: When we include requestId in the SELECT clause on the V2 multi-stage engine (/query), the query gets cancelled immediately with EMPTY_MAILBOX_SEND / error 503 CANCELLED errors. Omitting requestId from SELECT works perfectly — cursor, pagination, everything functions as expected. We believe this is a namespace collision between V2's internal gRPC stage-tracking (which uses requestId for inter-stage communication) and our column named requestId. requestId is our primary key — we cannot omit it from SELECT. We need it for record-level validation between data sources. We are thinking of creating new column with changed name since pinot columns are immutable , was considering using transform function to get old data of requestId column in new column. Just needed an opinion / heads-up on this approach.
j
Can you please share the entire error?
r
Copy code
[PINOT-DEBUG] raw response: {"numRowsResultSet":0,"partialResult":true,"exceptions":[{"message":"Received 1 error: Cancelled by sender with exception: CANCELLED: client cancelled","errorCode":503}],"numGroupsLimitReached":false,"numGroupsWarningLimitReached":false,"maxRowsInJoinReached":false,"maxRowsInWindowReached":false,"timeUsedMs":1946,"stageStats":{"type":"MAILBOX_RECEIVE","parallelism":1,"clockTimeMs":0,"children":[{"type":"MAILBOX_SEND","stage":1,"parallelism":1,"fanOut":1,"rawMessages":2,"serializedBytes":22708835,"serializationTimeMs":109,"clockTimeMs":0,"children":[{"type":"SORT_OR_LIMIT","executionTimeMs":486,"emittedRows":293645,"parallelism":1,"clockTimeMs":486,"children":[{"type":"MAILBOX_RECEIVE","executionTimeMs":447,"emittedRows":293645,"fanIn":90,"inMemoryMessages":1,"rawMessages":122,"deserializedBytes":44869955,"deserializationTimeMs":36,"upstreamWaitMs":8414,"parallelism":1,"clockTimeMs":447,"children":[{"type":"EMPTY_MAILBOX_SEND","stage":2,"description":"No stats available for this stage","table":"URDMCore"}]}]}]}]},"maxRowsInOperator":293645,"requestId":"1832875796000000000","clientRequestId":null,"brokerId":"Broker_pinot-broker-3.pinot-broker-headless.reolaps-pinot-cluster-1.svc.cluster.local_8099","numDocsScanned":0,"totalDocs":0,"numEntriesScannedInFilter":0,"numEntriesScannedPostFilter":0,"numServersQueried":90,"numServersResponded":90,"numSegmentsQueried":0,"numSegmentsProcessed":0,"numSegmentsMatched":0,"numConsumingSegmentsQueried":0,"numConsumingSegmentsProcessed":0,"numConsumingSegmentsMatched":0,"minConsumingFreshnessTimeMs":0,"numSegmentsPrunedByBroker":0,"numSegmentsPrunedByServer":0,"numSegmentsPrunedInvalid":0,"numSegmentsPrunedByLimit":0,"numSegmentsPrunedByValue":0,"brokerReduceTimeMs":589,"offlineThreadCpuTimeNs":0,"realtimeThreadCpuTimeNs":0,"offlineSystemActivitiesCpuTimeNs":0,"realtimeSystemActivitiesCpuTimeNs":0,"offlineResponseSerializationCpuTimeNs":0,"realtimeResponseSerializationCpuTimeNs":0,"offlineTotalCpuTimeNs":0,"realtimeTotalCpuTimeNs":0,"explainPlanNumEmptyFilterSegments":0,"explainPlanNumMatchAllFilterSegments":0,"traceInfo":{},"tablesQueried":["URDMCore"],"offlineThreadMemAllocatedBytes":0,"realtimeThreadMemAllocatedBytes":0,"offlineResponseSerMemAllocatedBytes":0,"realtimeResponseSerMemAllocatedBytes":0,"offlineTotalMemAllocatedBytes":0,"realtimeTotalMemAllocatedBytes":0,"pools":[],"rlsFiltersApplied":false,"groupsTrimmed":false}
Did not receive any response , this is a genuine issue which we are facing with pinot . we tried introducing new column say x with transform function which would copy values of requestId column to that new column . Data started persisting restarted segments , old values got copied still facing this issue even while querying with that new column name.
Copy code
"exceptions":[{"message":"Received 1 error: Cancelled by sender with exception: CANCELLED: client cancelled","errorCode":503}]
j
I'll take a look at this today
What's the data type of requestId column? Did you try running the same query using Pinot Query UI?
I don't suspect 
requestId
being reserved or something causing the issues
r
"dataType": "STRING",
j
What's the exact query if you can share?
r
yes , but does ui support pagination ? such large records we were unable to fetch in a paginated manner from ui , like 100000- 3000000
Copy code
Pinot JSON payload: {"sql":"SELECT urdmCore.urdmId, urdmCore.s3MetaData FROM urdmCore WHERE urdmCore.evaluationType = 'LIVE' AND urdmCore.intentId = 'jp_physical_inline_buyer_fraud_check_checkout' AND urdmCore.evaluationStatus = 'SUCCESS' AND urdmCore.eventTimestamp BETWEEN fromDateTime('2026-04-19 01:45:00', 'yyyy-MM-dd HH:mm:ss') AND fromDateTime('2026-04-19 01:50:00', 'yyyy-MM-dd HH:mm:ss') LIMIT 1000000","queryOptions":"getCursor=true;cursorNumRows=10000;timeoutMs=60000"}
j
I wanted to rule out MSE being the issue. If MSE is an issue, then UI queries will fail too
Can you try with a smaller limit once?
r
paginated queries cannot be fired the same way through ui right ?
we were trying with 50k per page then reduced to 10k .
j
Yes you cannot run cursor queries from UI.
we were trying with 50k per page then reduced to 10k .
I meant reducing the original query limit
Just try fetching a single row
r
first page we are able to fetch even with this limit , it is the paginated responses for which we are getting above error.
j
btw, why are you using MSE for this query?
r
tried Single stage engine as well, but we were issues like results not present , again for paginated responses not first page , so after understanding the details on how single stage engine works - correct me if I am wrong, In single stage engine broker stores the entire result of a query in its local memory and then we can fetch in a paginated manner
but our architecture is such that we have nlb and (network load balancer) it does not support sticky routing.
so basically for single stage we need to query and fire subsequest queries to that very broker which is not possible for us
j
If the same query fails in SSE, then it's a setup issue and not a MSE specific issue. Cursor code is the same for SSE or MSE.
r
issue is separate there .
j
You should try isolating the issue first. Try running the cursor query once with the broker host + port and fetching the subsequent results from the same broker for both SSE and MSE. If those do not fail, we can rule out Pinot side issues.
r
it is not possible to fire using broker host and port since we are firing via lambda , lambda is the client ( it is in a separate vpc) .
j
Can't you bypass lambda for testing purposes?
r
so it is working for less tps use case , this issue still coming for high tps use case , now pagination might not be the root cause but the issue thrown is very generic .
j
I strongly suspect the main issue is query timeout. For large limit query, can you increase the timeout and try?
r
yes sure.
Checked broker logs , RESOURCE_EXHAUSTED: gRPC message exceeds maximum size 16777216: 17391545 RESOURCE_EXHAUSTED: gRPC message exceeds maximum size 16777216: 19425988 RESOURCE_EXHAUSTED: gRPC message exceeds maximum size 16777216: 21476562 found the above issue , is there a way to increase this size limit from our end ?
j
@Gonzalo Ortiz / @Yash Mayya I'm not fully aware of the the MSE configs. Is 
pinot.query.runner.max.msg.size.bytes
the correct config to set?
y
Yes, that's the right config. @Richa Kumari what version are you running? IIRC this was an issue affecting older versions, but shouldn't be an issue in the latest release (Pinot 1.5.0)
👍 1
r
Changing the size limit worked pinot.query.runner.max.msg.size.bytes=268435456 pinot.broker.grpc.max.inbound.message.size=268435456 Thanx!
@Yash Mayya 1.4.0 What do you mean when u say this shouldn't be an issue in the latest release ? as in there is no size limit ?
y
Blocks are automatically split to avoid this issue
r
Okay, Thanx !
9 Views
Open in Slack
PreviousNext
Powered by