can I do pagination using presto and pinot on the ...
# generalk
kant
07/11/2020, 10:49 PMcan I do pagination using presto and pinot on the joined result set? for example can I get 10K rows at a time up to say 1M? Currently we use ES and it display only 10K rows(without joins) by default and anything beyond that there is scroll API however it doesn't quite work well and takes a lot time
x
Xiang Fu
07/12/2020, 9:29 PM
This is not supported yet, presto has to fetch all the data at once. We are working on support streaming APIs in Pinot right now, once that is in place, we can update presto to support better paging
k
kant
07/13/2020, 11:16 PMgot it! Looking forward
kant
07/15/2020, 7:28 AMhttps://github.com/prestodb/presto/issues/7636 Looks like Pagination is not supported in prestodb..so I guess need to call pinot directly for pagination?
kant
07/15/2020, 7:40 AMhttps://prestosql.io/blog/2020/02/03/beyond-limit-presto-meets-offset-and-ties.html This looks very recent but glad it's there
x
Xiang Fu
07/15/2020, 10:47 AM
Pinot streaming server will provide the grpc apis to get records in batches. we plan to use this in the presto pinot connector to get results from pinot