Hi All, We want to run compaction manually instead of the automated mode. This is mainly to do with availability of workers to run the compaction tasks. We tested by submitting a simple "compaction" task to the "task" API keeping most of the defaults. Example: { "type": "compact", "datasource": "datasource", "ioConfig": { "type": "compact", "inputSpec": { "type": "interval", "interval": "yyyy-MM-dd/yyyy-MM-dd" }, "dropExisting": true } } However we noticed that the compacted segments no longer retain the partitioning scheme that was originally used. Originally the segments were "hash" partitioned and newly written segments used "dynamic" partitioning. Doesn't compaction retain the parameters used to create the original segments - partitioning scheme, rowsPerSegment, clustering, order of columns etc ? Thanks, AR.

Since data can be incrementally ingested for a table with different partitioning configurations, druid doesn't attempt to infer it from the segments being compacted. It uses

dynamic

by default. To override that in the manual compaction job, refer to

partitionsSpec

inside the

tuningConfig

param for manual compaction

Thanks Vishesh. I did go through the docs and understand that we can specify

partitionSpec

etc. But that should be in cases where we want to change the parameters from the original ingestion. It defeats the purpose of having a separate task type if we need to specify all the details once more when submitting the compaction task. We now need to remember which params will be retained and which of them will change from the original segment. This adds to complexity if the original ingestion used the MSQ engine (as in our case) while compaction uses a JSON spec. We could as well perform a reindex instead of compaction in that case (which is what we are planning to do) since we can reuse the original ingestion params and just change the source to the Druid table itself. Thanks, AR.

I understand. The reason for not inferring it, as I said, is the possibility of segments being ingested with different partitioning schemes. Everything else in compaction spec need not be specified and can be calculated at runtime; a way to figure that out is to look at the fields that are explicitly populated by default when creating an auto-compaction spec in the web-console. That said, we do have partitioning info available per-segment which could be used if its value is unanimous across segments being compacted. You can file a proposal for such a feature.

Hi Vishesh, What happens in the case of auto-compaction? Does it retain the partitioning and other configs that were set during the original ingestion? Thanks, AR.

No it doesn't. Auto differs from manual compaction only in the scheduling aspect.