Hi Team, has anyone considered the idea of having a separate Middle-manager to perform compaction? The idea is that compaction will be performed by a separate/predefined middle manager, so that we can segregate the normal running tasks like querying and ingestion from compaction. any leads would be really helpful.

Yes, you can separate out any tasks to specific tiers using the Overlord Dynamic config: https://druid.apache.org/docs/latest/configuration/#overlord-dynamic-configuration I have routinely carved out an ingestion-only tier to handle high volume real-time queries. Carving out a separate tier just for compaction can also help alleviate congestion with other tasks.

Thanks John, will try this out.