question… what happens if a really old event gets thru a realtime table? wonder what the implications of it are in a hybrid setup, would that record be lost in the ether?

If it is older than time boundary, it won’t be served from realtime.

and offline will also don’t get it right

another option is to keep 2 columns: • timeColumn is the actual event time • another datetime column is the time you want to filter by

All rows ingested (with whatever timestamp) will be available for query. Old segments are removed periodically if the newest data in that segment is past retention period. So, for example, your retention is 7d and you ingest one row at 5am that is 10d old but all other rows are in good time, then the old row will be available for 7d

thank you so much and oh interesting, and all things would work as usual? what if i also have a realtimetooffline task to move to offline server

I am not 100% sure if the rt2offline task filters out the expired rows.

anyone i should ask that would know this?

Luis, one option would be to use stream processing to dead letter queue any data older than X days.

Another is to include in your query a where clause that filters out anything older than your retention time

If using r2o, that late event won’t be transferred to OFFLINE if the watermark has passed it. See this limitation for more details: https://docs.pinot.apache.org/operators/operating-pinot/pinot-managed-offline-flows#late-data-problem. The solution in this case, if you cannot extend retention of realtime table and increase bufferTime for r2o job, is to push that late event as a segment manually to the offline table

thank you so much for all your suggestions 🙏 I think that we will go with having only a realtime table as it seems the easier thing to do