I have a highly aggregated real-time table that I’...
# troubleshooting
k
I have a highly aggregated real-time table that I’m using to query and chart statistics. Although I’ve added around 10 billion events, they’re aggregated(upserted) into about 500,000 rows. Despite this, the table currently takes up around 200 GB of storage. However, if I export the entire table using
SELECT * FROM table
and then re-import it using a simple tool, the size drops to just 15 MB. I only need the aggregated data — I don’t need per-event details. Is there a way to merge the old segments and significantly reduce table size and improve query speed using Pinot tasks?
m
There’s a compaction task that can delete the obsolete upserted rows, which in your case seems to be most of the data
k
@Mayank Thanks for the feedback! Are you referring to this task? 🔗 https://docs.pinot.apache.org/manage-data/data-import/upsert-and-dedup/segment-compaction-on-upserts I also came across this GitHub issue: 🔗 https://github.com/apache/pinot/issues/14305 I noticed it mentions the
UpsertCompactMergeTask
, but I couldn’t find any public documentation for it. Would you mind sharing the documentation if it’s available? Which is the earliest Apache Pinot version that supports the
UpsertCompactMergeTask
?
m
Yes that’s the feature. It has been there for quite sometime, so if you use the latest image, you should be good.
k
I’m not sure how to enable segment merging. The documentation above only covers compaction, but doesn’t explain how to actually perform merging. What am I missing? How can I enable segment merging in Pinot?
m
cc @Manish
k
@Mayank My understanding of this PR — https://github.com/apache/pinot/pull/14477 — is that there is a
CompactMergeTask
available. However, I couldn’t find any official documentation about it. Is there any public documentation or guidance on how to use it? Do I miss something else?
@Mayank by the way thank you for the quick and helpful response.
m
@Ankit Sultana tagging you as you seem to be the author of that PR
a
The feature doesn't work in Pinot 1.3 but it should work on master after the recent fixes from Tarun. @Tarun Mavani: can you help add documentation for this feature in the OSS Docs?
If it helps, Pinot 1.4 should release next month afaik and it should have support for this task.
k
Thank you everyone. It was very very helpful!
t
I will add the documentation very soon.
k
@Mayank ’m reading the documentation for the
MinionMergeRollupTask
, and it states that it won’t work on real-time tables with upserts enabled. My table is exactly that — a real-time table with upserts. Is there currently any task that can merge segments for this kind of table? @Ankit Sultana @Tarun Mavani — Will the new
CompactMergeTask
expected in Pinot 1.4 support real-time tables with upserts?
t
Yes. The
CompactMergeTask
is built to support merging segments for realtime upsert table. The task was added as part of the https://github.com/apache/pinot/pull/14477 + some follow ups. Here is the design doc if you are interested. I will raise the PR for the documentation for this task soon. In the meantime, if you have any question, feel free to ask me.
k
@Tarun Mavani Thank you very much!
t