Hello Everyone, I am working towards a project to ingest historic data to pinot. We have nearly 50 Peta bytes of data stored in S3. We have to process that data to extract metadata and store it in pinot tables. Previously we were publishing to kafka and pinot was reading from the kafka. Now, since this is historic data , we want to move the flow away from real time streaming. I have read about the batch ingestion via minions and spark jobs in the documentation. I want to know which of those serves better for our case considering the huge amount of data we have. Any suggestions and if possible providing any references by anyone will be much helpful to my work.

if you already have a spark cluster and this is one-time job, then do spark way