Hi Team, just wanted to ask a very basic question in Pinot 1.5 version do we have a way to manage table retention on a dimension/categorical field other than time based retention that we use today in our pinot 1.3 deployments. example - we have event data coming into our tables via kafka and we have Hybrid tables. with 7 Days RT and 30 Days OF line table. we have a lifecycle for each event at end of which status is Clear State we would like to drop all events that are in cleared state as this table we want to maintain only active events.

purgeTask may be the right way 😉

thank you let me read about it

purgeTask has no built-in filter, so you'd write a RecordPurger that drops rows where status is Cleared and register it in a custom minion starter. It works on completed segments in both the realtime and offline tables.