Hi just need a confirmation. When i created my tables, my tables did not have any data but segments were created. I checked my s3 and there was no segment uploaded. However, as soon as data arrived in my tables, I checked and saw that segments were then uploaded to S3. So, wanted to confirm if segments are uploaded only when it has some data ??
03/03/2022, 10:09 AM
Do you mean that you're using s3 as the deep store?
If so then for real-time tables you'd only see segments in s3 once they have been completed/flushed. The consuming segment(s) live only on Pinot servers until they reach segment threshold.
If it's an offline table then the job that you wrote to create and upload the segments would be putting those segments into s3.
In either case you are correct that creating a table doesn't upload segments to s3.
03/03/2022, 2:18 PM
Are you asking about the case where segments don’t have any data (no rows)? If so, I can double check but I think we did prevent pushing if empty segments a while back. You can also confirm it in controller log, that should indicate why the segment was not accepted
03/04/2022, 9:06 AM
@User yes, i am using s3 as deep store and tables are realtime.
@User yes, segements don't have any data and they were not pushded to s3. When data came, only then, I could see segements pushed to s3