Kiril Kalchev
06/23/2025, 11:37 AMSELECT * FROM table and then re-import it using a simple tool, the size drops to just 15 MB.
I only need the aggregated data — I don’t need per-event details.
Is there a way to merge the old segments and significantly reduce table size and improve query speed using Pinot tasks?Mayank
Kiril Kalchev
06/23/2025, 5:26 PMUpsertCompactMergeTask, but I couldn’t find any public documentation for it.
Would you mind sharing the documentation if it’s available? Which is the earliest Apache Pinot version that supports the UpsertCompactMergeTask?Mayank
Kiril Kalchev
06/23/2025, 6:39 PMMayank
Mayank
Kiril Kalchev
06/23/2025, 6:47 PMCompactMergeTask available.
However, I couldn’t find any official documentation about it.
Is there any public documentation or guidance on how to use it? Do I miss something else?Kiril Kalchev
06/23/2025, 6:59 PMMayank
Ankit Sultana
06/23/2025, 7:24 PMAnkit Sultana
06/23/2025, 7:25 PMKiril Kalchev
06/23/2025, 8:40 PMTarun Mavani
06/24/2025, 4:21 AMKiril Kalchev
06/24/2025, 8:37 AMMinionMergeRollupTask, and it states that it won’t work on real-time tables with upserts enabled. My table is exactly that — a real-time table with upserts.
Is there currently any task that can merge segments for this kind of table?
@Ankit Sultana @Tarun Mavani — Will the new CompactMergeTask expected in Pinot 1.4 support real-time tables with upserts?Tarun Mavani
06/24/2025, 8:42 AMCompactMergeTask is built to support merging segments for realtime upsert table.
The task was added as part of the https://github.com/apache/pinot/pull/14477 + some follow ups.
Here is the design doc if you are interested.
I will raise the PR for the documentation for this task soon. In the meantime, if you have any question, feel free to ask me.Kiril Kalchev
06/24/2025, 9:30 AMTarun Mavani
06/24/2025, 1:46 PM