Hey all! Looking for input on how to prevent compaction tasks from publishing overlapping segments. Here's what happened: • Auto-compaction was running every 30 minutes, compacting daily-partitioned data. • A manual compaction reindexed the same period with a different granularity. • Once the manual compaction finished, some previously scheduled auto-compaction tasks (created before the reindexing) started running. • These tasks compacted data with the old granularity, causing a mix of granularities and overlapping segments (daily segments were created inside of monthly one). • After the data was successfully compacted, things stabilized. For a fix, we need to prevent autocompaction from publishing overlapping segments. We know which segments will be published only at the InputSourceProcessor level on the worker. The indexing task has a

coordinator-issued

prefix, which helps identify auto-compaction and could be used to check and reject publishing segments that don’t match the current cluster state. The problem is that this prefix isn’t accessible on the level where segments are being chosen. The options are to add some preprocessing of the input source at generateAndPublishSegments in IndexTask or to send some flag to the InputSourceProcessor.prosess() method, but both feel like a last resort. Curious to hear your thoughts — if not there, where do you think this could be best handled?

For auto-compaction, I'd be looking further upstream, to prevent the task from being issued at all

There is also compactionState which can be used in the replace command which prevents the compactSegments duty to issue those tasks

Thanks for the response! Although it seems to me that since auto-compaction tasks for a given interval were already queued before the manual compaction ran, it means they were valid from the Supervisor perspective. Once the manual compaction finished, those waiting auto-compactions ran, and re-compacted the same intervals at the old granularity, creating overlapping daily-in-monthly segments. I see two options to prevent this issue: • Prune pending tasks: Whenever you do a manual compaction, cancel any queued auto-compactions that target those same intervals. • Double-check before publish: Right before publishing, have each compaction task fetch the current segments and validate if it's going to publish overlapping segments.