Hi, Looking to understand the current capabilities...
# generalg
Grace Walkuski
10/17/2022, 4:47 PMHi, Looking to understand the current capabilities around getting large amounts of data out of Pinot in some sort of chunks. The docs are hard to sort through. I’d need any of these supported with aggregation. Can it stream? page? shard? If not now, is in the works soon? tia
g
Ganesh
10/17/2022, 4:53 PMYou can use Apache Spark to read data from Pinot. There is a connector available with documentation https://github.com/apache/pinot/tree/master/pinot-connectors/pinot-spark-connector
g
Grace Walkuski
10/17/2022, 4:59 PM@Susan Candela
k
Kishore G
10/17/2022, 5:41 PM
Suggest breaking it by time.. there is also an hidden feature where you can restrict the query to a specific segment as well
g
Grace Walkuski
10/17/2022, 5:54 PMThat could be an option. We were hoping to find one query pattern/solution that would work for both elasticsearch and pinot
k
Kishore G
10/17/2022, 6:36 PM
splitting it by time range..
Kishore G
10/17/2022, 6:39 PM
I would do this in multiple steps
• select count(*) from T where time between t1 and t2
• If the count is large > Threshold- break up the query into multiple time ranges.. you can assume that data is evenly spread across each time range
• else count is small < Threshold - send one query
g
Grace Walkuski
10/17/2022, 6:45 PMOk thanks
x
Xiang Fu
10/17/2022, 10:15 PM
Another option is to use presto/trino on pinot for streaming data back.