As far as I can see I can use built-in purge task. The only thing I need to do is to setRecordPurgeFactory in MinionContext. Can’t figure out how

Hi. This have to be done in the minonBaseStarter.

SegmentPurger.RecordPurgerFactory s = new SegmentPurger.RecordPurgerFactory() {
  @Override
  public SegmentPurger.RecordPurger getRecordPurger(String rawTableName) {
    _listToPurge = getMapFromCSV(System.getenv("PURGE_LIST_FILE"));
    SegmentPurger.RecordPurger r = row -> {
      String contextTable = rawTableName.replace("_OFFLINE", "");
      for (List<String> a : _listToPurge) {
        if (a.get(0).equals(contextTable)) {
            if (a.get(1).equals(row.getValue("meta.customer"))
                    && (((a.get(2).equals("*"))
                    || (a.get(2).equals(row.getValue("id")))))) {
                return true;
            }
        }
      }
      return false;
    };
    return r;
  }
};
 minionContext.setRecordPurgerFactory(s);

What I’ve implemented in my side 😉 Many way to do it in a différent ways for sure

Thank you! I read through the thread. I hoped minion tasks could be setup through configs. I’m ok providing a jar with the right classes. I’m not looking to modify actual pinot code base and deploy a custom version.

This should work. What is you pinot version ? Latest master ?

Well. I have Ops who install and configure clusters, etc. Don’t want to customize source code. Plug-ins is a fair game.

You will have to dev something already tested and dev in other version. Not sure the plugin thing will work @Mayank any point of view to share on this ?

I understood that