Is there a cluster config to periodically clean up the task history to avoid bogging down ZK? I know there's an API, just wanted to know if it could be self contained without having to schedule an job external to pinot to call it.

There doesn't seem to be a periodic task for this. cc: @Manish @Shounak Kulkarni

Could look into implementing this as well if deemed necessary

Awesome, thanks 🙏

@Tommaso Peresson - Are you referring to the completed tasks (minion task/subtask and their configs) stored in Zk? Helix has

TaskGarbageCollectionStage

that looks for completed/expired/dangling tasks and cleans them up periodically.

Yes, should I look at the code or do you know how to set the frequency of garbage collection?

This is helix code. Default expiry seems 24 hrs. Code that uses expiry

public static final long DEFAULT_EXPIRY = 24 * 60 * 60 * 1000;

and there's workflow config param called `Expiry` Have to dig in to see how to set this property. I think this will allow us to have control over when its cleaned up

public enum WorkflowConfigProperty {
  WorkflowID,
  Dag,
  ParallelJobs,
  TargetState,
  Expiry,
  StartTime,
  RecurrenceUnit,
  RecurrenceInterval,
  Terminable,
  FailureThreshold,
  /* this is for non-terminable workflow. */
  capacity,
  WorkflowType,
  JobTypes,
  IsJobQueue,
  JobPurgeInterval,
  /* Allow multiple jobs in this workflow to be assigned to a same instance or not */
  AllowOverlapJobAssignment,
  Timeout
}

Thanks I'll have a look to see if it's configurable now, otherwise I'll make it