Is it possible to control the TTL of the keys kept in memory for Pinot deduplication? i.e. My table's retention is set to 90 days, but dupes are only possible for the initial 24 hours. I'd like to enable deduplication for the first 24 hours so I don't have to pay the price of keeping the keys of 90 days worth of data in memory.

I think we were discussing this at some point. Is it in the plan @Jackie @Kartik Khare?

Also does dedup work on hybrid tables? If so would moving the segments from realtime to offline remove the primary keys from memory?

@saurabh dubey for dedup. it doesn’t work on offline tables. so moving segments from r2o would only keep pks for the realtime part

So it should work if you keep low retention for real-time table and move segments to offline table

perfect, thanks!