https://pinot.apache.org/ logo
#general
Title
# general
i

Itzik Lavon

08/08/2020, 9:09 AM
Hi all i'm new to this is there any good videos on how to gather data from S3 and query it? another question, currently i use Athena to do the queries this is better? sorry for my ignorance 🙂
o

Oguzhan Mangir

08/08/2020, 9:31 AM
For pinot, currently, you can analyze data with your favorite tool like spark, hive etc then write it to the s3. Then you can trigger pinot spark-batch-ingestion spec(https://docs.pinot.apache.org/basics/data-import/batch-ingestion/spark) to create pinot segments and upload them to pinot. Then you can query data in ui or over http. I do not use athena, but athena queries s3 data with sql query. But pinot is designed for real-time olap queries with some other features. The use cases are different for pinot and athena i think. If I am wrong, we can talk about them
k

Kartik Khare

08/08/2020, 9:33 AM
https://docs.pinot.apache.org/basics/data-import/pinot-file-system/amazon-s3 This should help. Let me know if you require more details
i

Itzik Lavon

08/08/2020, 9:38 AM
@Kartik Khare thanks for the reply i've read it, but didn't understand on how to use it when deploying via docker
i just want it to import the data which is already there
k

Kartik Khare

08/08/2020, 9:49 AM
You should just mention the file spec in the job spec e.g.
Copy code
pinotFSSpecs:
  - scheme: s3
    className: org.apache.pinot.plugin.filesystem.S3PinotFS