Hello ! 👋 I've got the following scenario : • Data is integrated in multiple batches per day (in an OFFLINE table) ◦ Batch 1: _01/01/2021 (data date) - DATA 1, DATA 3, DATA 6 -> `Segment_1(date=01/01/2021, data=[DATA 1, DATA 3, DATA 6])`_ ◦ Batch 2: _01/01/2021 (data date) - DATA 2, DATA 4, DATA 5 -> `Segment_2(date=01/01/2021, data=[DATA 2, DATA 4, DATA 5])`_ • Data must be available asap, so 2 separate segments are generated & ingested into Pinot • Some data needs to be corrected after the initial data ingestion, say DATA 1 & DATA 2 I know it is possible to replace segments but, how can we handle replacing data across multiple segments ? Can we generate a new segment with only the modified data and ignore old data in previous segments ? (

Segment_1

&

Segment_2

) ->

Segment_3(date=01/01/2021, data=[DATA 1, DATA 2])

Or do we have to regenerate the 2 segments entirely ? (if so, we need to identify what they contain) - Possibly after merging them ?

You can regenerate the two segments (using same name as existing segments) and push them to Pinot. Currently this is not an atomic transaction so there may be some small time period when one segment is old and another is new. This is being worked on to fix. @Seunghyun

Thanks @Mayank So it is necessary to know the contents of the 2 segments and regenerate them with the same data as before (+ updates) ? Sounds like this could be non trivial in some cases

Pinot won’t know that it is duplicate data, and will be included in query processing

If you are generating daily segments then replacing one days segments should be straight forward

The difficulty is that not all day's data may / will arrive at the same time but ingestion Hence • Batch 1:   _01/01/2021 (data date) - DATA 1, DATA 3, DATA 6   -> `Segment_1(date=01/01/2021, data=[DATA 1, DATA 3, DATA 6])`_ • Batch 2:   _01/01/2021 (data date) - DATA 2, DATA 4, DATA 5  -> `Segment_2(date=01/01/2021, data=[DATA 2, DATA 4, DATA 5])`_ In the end, I feel like my question is "how can we update part of a segment" ? I feel like it's not possible then It looks like there's only 2 ways to reach my goal then : 1. Only have a single segment per day at a time so a. Drop day's segment b. Regenerate segment with updated data [99% of the data may not have changed, so pretty inefficient] 2. Identify impacted segments & regenerate impacted one only (in their entirety) What do you think ? 🙂

Is your offline pipeline not generating daily partitions? Typically offline pipelines would created time partitioned folders, and generating segment from one folder will guarantee to not overlap with other days

It is but we have 3 additional constraints • Data for a given day can arrive in multiple parts (for the same day) [imagine the case with N timezones} • Partial data need to be available asap (can't wait for other parts) • Need to be able to update some data later on (doesn't need to be perfectly efficient, as it's clearly not an ideal case for OLAP)

Do you not have realtime component? If you do then you can serve data from realtime while your offline settles?

I feel like this would help, but no, data comes in batches from external sources.. I'll keep that in mind still

What is the max delay for data to arrive? Does one day's worth of data settle in a day or so? Or it can take several days / weeks?

What is the max delay for data to arrive? Does one day's worth of data settle in a day or so? Or it can take several days / weeks?

Typically much less than a month but it is technically unbounded - customer data could theoretically be corrected months after first ingestion

When it arrives after a month, which folder does it land in? Is it in the correct date folder? Also, how do you know which older folders got changed?

Also, even if your incoming data is not partitioned, you can always generate segments to guarantee the data belongs to one day (eg pick several folders to scan and select data only for single day to generate input for pinot segment)

If I understand correctly, you're saying that after every batch, we generate the whole pinot segment ? For example, we've got a single file per batch and after every batch, we could regenerate a single pinot segment from every one of these files Meaning we always keep a single Pinot segment (per day) at a time, and replacing it is straightforward

Discussed offline, @Jonathan Meyer to summarize.

Yes 🙂

Yes, thanks

> a new feature will allow generating a single Pinot segment out of multiple input files, dropping the need for file concatenation

Is there any updates to this? am looking for something similar