Hi team, my hybrid table is designed to push 5 day...
# general
p
Hi team, my hybrid table is designed to push 5 days old data from realtime to offline. But I need to backfill 2 days old data. Can I delete the realtime segment and backfill 2 days old data directly as an offline segment? Will that create any side-effects or minion exceptions?
k
Yes.. it should work.. you don’t have to delete real-time data
Just push new data to offline table
Oh forgot to ask, Are you using minion to move data from real-time to offline?
p
Hi @Kishore G, yes there is a minion task that runs everyday to push realtime to offline. The 2 days old data has some wrongness to it. So should I wait for minion to push it to offline and then drop that segment, or can I drop the segment in realtime and backfill in offline directly? I guess one of the concerns is will the minion continue working un-interrupted the next day even if it finds 1 day’s data missing in realtime table
k
Yeah.. it requires some careful orchestration