https://pinot.apache.org/ logo
#general
Title
# general
p

Prakash Tirumalareddy

10/01/2020, 3:44 AM
Hello Everyone, Very quick question I have lots of data in s3 bucket (parquet format). Can I use Pinot to retrieve or query single record base on query or condition? Is Pinot right software to do such thing?
m

Mayank

10/01/2020, 3:57 AM
Hello, is your requirement just to access the data, or do you want to run some analytical queries?
p

Prakash Tirumalareddy

10/01/2020, 3:58 AM
Hello Mayank, both
m

Mayank

10/01/2020, 3:58 AM
Ok, for analytics, Pinot is certainly the right software to do it
The data on S3 can be batch ingested into Pinot, and then can be queried
p

Prakash Tirumalareddy

10/01/2020, 4:01 AM
what about query to specific data? is it possible fastest way possible?
m

Mayank

10/01/2020, 4:01 AM
what do you mean by query to specific data?
You want to run a query directly on the data sitting on S3?
p

Prakash Tirumalareddy

10/01/2020, 4:03 AM
yes (don't want to use Athena because it is slow)
m

Mayank

10/01/2020, 4:03 AM
Hmm, ok, to use Pinot, the data has to be at least indexed and ingested into pinot
p

Prakash Tirumalareddy

10/01/2020, 4:04 AM
Yes I want to ingest the data to Pinot and then retireve via API or query..
m

Mayank

10/01/2020, 4:04 AM
You can explore Presto if you just need a SQL engine on top of data at rest
p

Prakash Tirumalareddy

10/01/2020, 4:07 AM
Yes idea is what you said: 1. S3 can be batch ingested into Pinot, and then can be queried 2. Query via API or using Pinot Dashboard for business analytics queries
m

Mayank

10/01/2020, 4:07 AM
The idea sounds good
p

Prakash Tirumalareddy

10/01/2020, 4:08 AM
Thank you very @Mayank I will start work it now 🙂
m

Mayank

10/01/2020, 4:08 AM
👍
p

Prakash Tirumalareddy

10/01/2020, 4:22 AM
Sorry one more question.. do you aware of any example or article to achieve the above?
m

Mayank

10/01/2020, 4:24 AM
That is for configuring S3 as the deep-store for Pinot
You can also look at the docs above for setting up batch ingestion
p

Prakash Tirumalareddy

10/01/2020, 4:27 AM
ok sure thank you..
Sorry again for asking so many question... is there any tool to create Pinot table & schema configuration json file from athena table or from glue?
or does this need to be done manually?
m

Mayank

10/01/2020, 4:32 AM
I am not aware if there is one
p

Prakash Tirumalareddy

10/01/2020, 4:32 AM
ok
k

Kishore G

10/01/2020, 4:42 AM
There is no tool to do that
p

Prakash Tirumalareddy

10/01/2020, 4:43 AM
ok thanks @Kishore G
ok I will write small tool from converting DDL to Pinot table schema json file using node for future use 🙂
👍 3
m

Mayank

10/01/2020, 4:48 AM
Awesome