A few related questions about Pinot ingesting data from a Google Cloud Bucket. Background: We were able to read in ~3 years of data from our production system that Pinot stored in 9800 ".avro" files (each is about 400 MB, so about 4 TB in total). That adds up to 2.5 billion transactions, and we were interested in measuring how the system would perform with 100 billion transactions present. Over this 3-day weekend, I created 39 copies of the .avro files (each with a different multiple of 3 years of a time shift), so that all 39 copies had unique values for the timestamp (there is just one dimension column - that time stamp - for the table in question). Over the weekend, my loop created new (gzipped) avro files with a for each record, and uploaded the 39 * 9800 ".avro.gz" files into our Google Cloud bucket that minion processes ingest from. We're only seeing a very small number of error messages, and yet very little of the generated data is present. Pinot is only seeing ~4.4 billion transactions (not even one full copy, much less 39 full copies) and it's not clear how to track down how far along the ingestion process is moving along (inside of Pinot) -- is there a way to see a list of still-pending .avro files (i.e. Pinot has seen that a new file was uploaded, but it hasn't started to import the contents yet)? (It may just be that I just don't yet know where to look.)

just want to double check, the ingestion is finished, right? nothing in progress?

this is a good idea; I will consult with my teammates to see if this will work for us.