https://pinot.apache.org/ logo
#general
Title
# general
p

Peter Pringle

08/25/2021, 11:22 AM
Can the pinot-admin MergeSegments command be used on real time segments which are online. I have a lot of 10mb segments and am tweaking the segment flush threshold time. Else guess I can try hybrid table to take the older ones into offline tables.
m

Mayank

08/25/2021, 5:23 PM
We are working on merge/rollup using minion cc @User
👍 1
j

Jackie

08/25/2021, 5:50 PM
You may use the realtime-to-offline task documented here to make it a hybrid table: https://docs.pinot.apache.org/operators/operating-pinot/pinot-managed-offline-flows
Once the merge/rollup task is available, you can enable it on the offline table to further merge the segments
p

Peter Pringle

08/26/2021, 1:14 AM
Think I read that upsert won't work for realtime segments converted to offline? Is this the case
For the new roll up feature would be neat if we could roll up the last value for a given primary key every x hours I.e. like upsert /latest aggregation operator in druid. This way I can reduce the resolution of my data over time.
j

Jackie

08/26/2021, 6:09 PM
Yes, upsert only works on realtime only table as of now. We are considering supporting it for hybrid table, and then rollup with only the last value would be needed in order to get the correct value for rolled-up time