Can the pinot-admin MergeSegments command be used ...
# generalp
Peter Pringle
08/25/2021, 11:22 AMCan the pinot-admin MergeSegments command be used on real time segments which are online. I have a lot of 10mb segments and am tweaking the segment flush threshold time. Else guess I can try hybrid table to take the older ones into offline tables.
m
Mayank
08/25/2021, 5:23 PM
We are working on merge/rollup using minion cc @User
👍 1
j
Jackie
08/25/2021, 5:50 PMYou may use the realtime-to-offline task documented here to make it a hybrid table: https://docs.pinot.apache.org/operators/operating-pinot/pinot-managed-offline-flows
Jackie
08/25/2021, 5:51 PMOnce the merge/rollup task is available, you can enable it on the offline table to further merge the segments
p
Peter Pringle
08/26/2021, 1:14 AMThink I read that upsert won't work for realtime segments converted to offline? Is this the case
Peter Pringle
08/26/2021, 1:19 AMFor the new roll up feature would be neat if we could roll up the last value for a given primary key every x hours I.e. like upsert /latest aggregation operator in druid. This way I can reduce the resolution of my data over time.
j
Jackie
08/26/2021, 6:09 PMYes, upsert only works on realtime only table as of now. We are considering supporting it for hybrid table, and then rollup with only the last value would be needed in order to get the correct value for rolled-up time