Hi team.. quick question: in a hybrid table, can we upload offline segments with overlapping time ranges but with unique names? We have a use case where we will need to backfill a hybrid table but the new data is not organized in time buckets but rather as a single csv spanning a large time range across existing segments. I'm wondering if we can simply create a new segment with the backfill data with unique name and upload it to the offline table and let mergeAndRollup job handle recreating segments with the correct ranges..

currently, we don't have the time based backfill that would clean up the old segments. Is it an option for you to upload the data & clean up the old segments?

Do I need to cleanup old segments even if the data is not duplicate but segments form overlapping time ranges? For example, I have two existing offline segments between with time ranges 2022-10-10_2022-10-15 and 2022-10-15_2022-10-20... If I upload a new segment with data between 2022-10-13_2022-10-17, do I need to do any cleanup? Or can I operate with these three segments and queries will work without any issues?

So, currently, Pinot’s data upload is based on

segment name

. If you upload segment with the same segment name, it will overwrite the data with the new one. If you upload segment with different name, it will be appended to the existing table.

Thanks for the explanation.. will double counting be a problem even if the data is different in the new segment and not repetitive?

then it should be fine

Great.. that should work for our use-case then.. thanks a lot @Seunghyun

👍

Ahh okay.. so it will stay independent and not get merged by the job.. but all the new data the gets ingested will continue to get merged and rolled up?

yes

Cool.. that should be fine for us given we expect these new segments to be limited

👍

Is there a plan to support backfill with merge and rollup in future?

yes but it will take some time. we are still in the design phase.

Okay.. I will keep an eye out for this feature.. thanks again!