Hi I'm exploring Pinot as a potential serving layer replacement of HDFS in our org.... Does writing orc files from s3 (5Tb in batches for now) everyday makes sense for Pinot?

5TB every day or 5TB in total

everyday retained upto 5 days

yeah, that should be fine

for more context: we're storing around 30Tb in total at any time in HDFS, tried replacing it with AWS Aurora (Postgres) though the reads are sub ms , writes have been a constant pain to deal with

make sure you use the metadata + uri based push

any place you'd personally recommend to begin with for this particular scenario for like a quick demo of sorts

@Raghav I am also curious to learn a bit more on what kind of querying capability are you looking for?