Does anyone have any good strategies for how they ...
# generalb
Brook E
09/30/2025, 3:29 PMDoes anyone have any good strategies for how they automatically toggle data from real-time to offline?
r
robert zych
09/30/2025, 3:32 PMb
Brook E
09/30/2025, 3:54 PMOne of the challenges we have is we have up-sert enabled. How would you recommend we handle that?
r
robert zych
09/30/2025, 4:10 PMupsert is only supported for real-time tables
robert zych
09/30/2025, 4:13 PMWhy are you looking to move from real-time to offline?
b
Brook E
09/30/2025, 4:36 PMWe are consuming real-time feeds from our operations which we want to then archive automatically after 30 days
Brook E
09/30/2025, 4:37 PMour goal is to do a rolling 18 month of analytics
Brook E
09/30/2025, 4:37 PMin that world, any data we consume should 'auto archive' so we can do same time last month / last quarter / last year analysis
m
Mayank
09/30/2025, 10:22 PM
How late can a record be upserted? If it is beyond the 30 days, then it breaks the upsert invariant of having all records of a PK on a single server
b
Brook E
09/30/2025, 10:54 PMdata cannot be upserted beyond 21 days
Brook E
09/30/2025, 10:54 PMOur goal is to automatically archive data every day as it 'ages-out'
m
Mayank
09/30/2025, 11:29 PM
In that case it might work, as long as you guarantee keys are not recycled, and also need to use compaction to remove obsolete records to be removed before aging out
b
Brook E
10/01/2025, 10:22 AMCool. So how would you recommend we achieve / do that?
m
Manish
10/01/2025, 3:57 PMI guess the requirement is to have upsert enabled realtime table and periodically move older data to offline table to do aggregation (weekly/monthly/yearly). So realtime gives the upsert support and offline is used for analytics ?
I dont think realtimeToOffline task that moves data from realtime to offline is supported on an upsert table
b
Brook E
10/01/2025, 3:58 PMbingo. that's exactly what we are trying to sort out