And if that's true, I'm not sure I even need my of...
# pinot-devs
Stuart Millholland
05/26/2022, 4:16 PMAnd if that's true, I'm not sure I even need my offline table
n
Neha Pawar
05/26/2022, 5:18 PM
yea doesn’t sound like you need the offline table. Some folks choose to move data to offline table (instead of being in the realtime table but on offline servers): https://docs.pinot.apache.org/operators/operating-pinot/pinot-managed-offline-flows
s
Stuart Millholland
05/26/2022, 5:20 PMThanks!
Stuart Millholland
05/26/2022, 5:26 PM@Neha Pawar one thing that is confusing me is that tiered storage and pinot offline managed flows seems to have a very similar outcome.
Stuart Millholland
05/26/2022, 5:27 PMIn tiered storage, I have just the realtime table but automatically move segments to offline servers after n number of days. I'm good with that. In pinot managed flows I would have both the realtime and offline tables and I could configure the offline movement to happen after n number of days.
Stuart Millholland
05/26/2022, 5:27 PMSo how do I know which option to go with?
Stuart Millholland
05/26/2022, 5:29 PMIt feels like the managed flow gives me more options, I can backfill data if needed into my offline table w/ that option I think. Any thoughts on those options and the pros and cons?
n
Neha Pawar
05/27/2022, 12:40 AM
how much is the table retention? generally better to have offline table if the retention is very large, as it becomes operationally easier if you have to do replace/upload/backfill etc. But downside is you need minions component added to your cluster
Neha Pawar
05/27/2022, 12:40 AM
one question I had, why do you even need to do this?
I have just the realtime table but automatically move segments to offline servers after n number of days.s
Stuart Millholland
05/27/2022, 1:28 PMSo we have 90 days retention and the idea is to keep 7 days in our realtime servers which are configured with much more ram so more $. We are ok w/ more latency as the data grows longer than 7 days.
n
Neha Pawar
05/27/2022, 3:29 PM
Got it. Then tiered makes more sense