Hi folks, what happens if one identical row exists in two different segments?(one is generated from realtime to offline flow while the other is from backfill)

If the records exists in multiple segments of an offline table, then you could get duplicated query results.

HYBRID table - If you already have a hybrid table this feature again may not be relevant to you. But you could explore using this to replace your offline push jobs, and simply keep them for backfills.

thanks. I came across this line - but it seems that we need to carefully coordinate these two jobs(RT2OFF and backfill) so as to not introduce duplications(e.g., when two time widows overlap)

That's correct.

🤔 how does it solve the duplication problem?

IIRC, the MergeRollupTask should replace segments in the realtime portion of your hybrid table instead of copying them to the offline portion where they may collide with your backfill process

In the case of hybrid tables, there is also a notion of

timeBoundary

that can prevent duplication strictly in cases where the same row exists in both REALTIME and OFFLINE counterpart tables, with the same value in the

timeColumn

column. A helpful example from StarTree docs[1]:

For example, if we executed the following query:

```SELECT count(*)

FROM events```

The broker would send the following query to the offline table:

```SELECT count(*)

FROM events_OFFLINE

WHERE timeColumn <= $timeBoundary```

And the following query to the real-time table:

```SELECT count(*)

FROM events_REALTIME

WHERE timeColumn > $timeBoundary```

The results of the two queries are merged by the broker before being returned to the client.

[1] https://dev.startree.ai/docs/pinot/concepts/time-boundary#querying

yeah in this case, the duplications happen to reside in offline table so the

timeBoundary

can’t really help here