@Xiang Fu @Mayank Record drops when querying a Realtime Pinot table via Trino Hi team, looking for some guidance on an issue reported by one of our tenants. • Pinot table:

managementReport

(Realtime) • Querying via Trino; queries fetch a large volume of data to take a snapshot and insert into Iceberg. • Queries include patterns such as regions, e.g.,

NY_SNAP_17.30

,

LN_SNAP_17.30

, etc. • When multiple regions are queried together, we see inconsistent counts for some regions. Querying each region individually returns consistent counts. • We noticed some Pinot servers intermittently return

0

records for certain requests, but there are no obvious errors in the Pinot logs, including

RESOURCE_EXHAUSTED

. • We added a retry mechanism in the Trino-Pinot connector, which resolves the issue and returns the expected count. Question: What would be the best way to debug this from the Pinot side and identify why and which server is returning 0 records? Are there specific Pinot metrics/logs or query-level diagnostics we should check? Has anyone seen similar behavior?

You probably are getting partial response flag in broker metadata because some servers may have not returned within time out. This implies query is very expensive, which is also confirmed by your statement. This is not a great way to use Pinot - ie bulk fetch data to insert into Iceberg:

queries fetch a large volume of data to take a snapshot and insert into Iceberg.

Can you tell a bit more about what is the business requirement, and what does your overall architecture look like?

The client queries Pinot from Trino via gRPC, so I’m assuming the broker may not be involved in this path. The query typically completes within 4–7 seconds; otherwise, Trino reports a timeout exception. What makes this difficult to troubleshoot is that in the cases we observed, the server returns 0 records without any apparent error. Our PROD architecture consists of 2 controllers, dedicated brokers, and 10 servers for this tenant. The client uses Realtime Pinot data with a 1-week retention. Since the data cannot be retained indefinitely in Pinot, they have a scheduled job that runs at different times for NAM, UK, and HK to fetch the data from Pinot and insert it into Iceberg for long-term EOD storage. Given this use case, are there recommended limits or best practices for the volume of data that can be bulk-fetched from Pinot via Trino? Also, is there a recommended approach for this type of EOD archival use case?

what's your trino query? do you have a time range with limit in the query?

Trino Query do not have a limit but the filter is with business date which is usually current date, and per region the record count is not more than 350k. The table retention is 5 days, total docs around 195 million.

Are you using Trino connector talking directly to server grpc? That should work for bulk data stream

Yes that's correct, we are using trino-pinot connector. So far we have added retry mechanism if record count is 0 but segment match has some values. Want to debug and understand further the reason for record drop, has anyone reported such issue?

The connector has two modes, one it talks to broker (default), second is where it directly talks to server. Latter is streaming/grpc based and is better suited for bulk downloads.

I am certain our use case is streaming gRPC-based. But is there a way to figure out the missing records?

Can you elaborate where you notice that? Is it the broker response, or server response or something else?

We noticed some Pinot servers intermittently return

0

records for certain requests, but there are no obvious errors in the Pinot logs, including

RESOURCE_EXHAUSTED

.

In the connector logs, we can see the number of segments queried, matched, and the documents returned. In some cases, the segment is processed and has matched segments, but the number of returned documents is

0

. For these scenarios, we’ve added a retry mechanism, which has helped us retrieve the expected results.

do you also modify trino connector code?

We added a check where, if

segmentsProcessed

and

segmentsMatched

are non-zero but

documentsReturned

is

0

, we retry the same query instead of returning an empty result. In some cases, the expected record count is returned after 2–3 retries.

Hmm, would grpc client configs tuning helps here?

We did add the above checks and retry mechanism. Would you recommend any other additional config which can help us to identify root cause and also fix the issue?

oh, I want to understand if this is client side issue or server side issue

Most likely the server didn't have a segment Trino asked for (it committed or moved after Trino planned the split). Pinot reports that as a SERVER_SEGMENT_MISSING exception in the final metadata block, but the Trino connector stops reading at that block without checking it, so you get 0 rows and no error. Check the numMissingSegments server metric for that table against your retry times. Filed it on the Trino side: github.com/trinodb/trino/issues/31888