We have an offline Pinot table with 20 columns experiencing high latency or timeout when query it with large offset: e.g. 1000000. some of the columns has large text: e.g. 50000 chars. Is there some thing we could change to help improving the query latency?

what does the query look like?

why query with large offset?

Typically Select * from table where timecolumname < somedate order by id limit 20 offset 1300000.

since it’s order by and offset, I think the broker will retrieve 1.3 million rows then give you 20

Yeah that make sense. But Is there anything we can optimize other than remove the ordering?

You want to use something with time

You mean ordering only by the field configured as timecolumnname?

Perhaps break query down into smaller queries for time ranges

sorry, got distracted and forgot to write the full solution