Hi Team, we are using

Apache Pinot 1.2.0

and we are facing a scaling issue with realtime upserts enabled for our use case, due to the nature of the primary key the updates can fall in after the expiration window of the

metadataTTL

defined (~8hrs) in our table config, since we cannot keep infinite retention so we are not able to show accurate information on our views due to duplicate entries being recorded We are thinking to write a custom minion job, which should do stuffs like 1. Fetch all such duplicate entries within a time window 2. Only keep the latest one and mark others as invalid so that they can be taken up for deletion in future We wanted to get community help as we didn't find any resources which provides such functionality open source , can anyone help with any references or starting point so that it can help us to use pinot efficiently cc: @Abhigyan Bharati @Xiang Fu

if you know the PK won't change after certain time, you can try a hybrid table setup, which you push all the immutable segments to offline table after certain time

The use case demands that the updates can still float in though in less quantity, consider if we ever had to do backfiling activity due to some kind of maintenance We where actually looking for a feasibility of writing a custom minion which can do such kind of deduping, so we wanted to know if that is even possible to consider it as one of the way for exploration and from where to start from

if you removed the PK then you will see duplicates when you backfill rows. There is a minion task for upsert merge: https://docs.pinot.apache.org/operators/operating-pinot/upsert-compact-merge-task