This message was deleted.

1. What’s the version of Druid ? 2. How many slots do you have on Indexers or MMs?

0.22.1, and 10 separate indexer services with 24 slots each

That was going to be next question…. compaction jobs take lower priority than the other jobs ( ingestion for example)

the task status is running so i think it’s all good there

Is your current compaction job automatic or manual ?

this one is manual, i’ve got it on automatic for most other datasources now but i wanted to do a test run on this datasource before turning it on there

Could you share your compaction job specs?

sure, it’s pretty basic. I submitted it as:

{
  "type": "compact",
  "dataSource": "xxx",
  "ioConfig": {
    "type": "compact",
    "inputSpec": {
      "type": "interval",
      "interval": "2023-01-31T16:00:00.000Z/2023-01-31T17:00:00.000Z"
    }
  }
}

and now it’s reporting the payload as:

{
  "type": "compact",
  "id": "compact_xxx_jfljkcjb_2023-02-03T00:52:28.068Z",
  "resource": {
    "availabilityGroup": "compact_xxx_jfljkcjb_2023-02-03T00:52:28.068Z",
    "requiredCapacity": 1
  },
  "dataSource": "xxx",
  "ioConfig": {
    "type": "compact",
    "inputSpec": {
      "type": "interval",
      "interval": "2023-01-31T16:00:00.000Z/2023-01-31T17:00:00.000Z",
      "sha256OfSortedSegmentIds": null
    },
    "dropExisting": false
  },
  "dimensionsSpec": null,
  "metricsSpec": null,
  "granularitySpec": null,
  "tuningConfig": null,
  "context": {
    "forceTimeChunkLock": true,
    "useLineageBasedSegmentAllocation": true
  },
  "groupId": "compact_xxx_jfljkcjb_2023-02-03T00:52:28.068Z"
}

One thing you can try is:

POST /druid/coordinator/v1/config/compaction/taskslots?ratio={someRatio}&max={someMaxSlots}

https://druid.apache.org/docs/latest/operations/api-reference.html#automatic-compaction-configuration It increases the capacity of the compaction tasks

Won’t that just allow me to do more concurrent compactions? That will help if each job takes several hours and I need to make sure i can be running enough jobs to keep up with ingestion, but I was wondering if there was a way to speed up a single compaction job somehow?

True …the

tuningConfig

is a parameter you can look into:

{
    "type": "compact",
    "id": <task_id>,
    "dataSource": <task_datasource>,
    "ioConfig": <IO config>,
    "dimensionsSpec": <custom dimensionsSpec>,
    "transformSpec": <custom transformSpec>,
    "metricsSpec": <custom metricsSpec>,
    "tuningConfig": <parallel indexing task tuningConfig>,
    "granularitySpec": <compaction task granularitySpec>,
    "context": <task context>
}

thanks - looks like i can try experimenting with maxNumConcurrentSubTasks, maxRowsInMemory, and maxBytesInMemory

Do let us know the outcome of your experimentation! Thanks

Maybe have a look at the

splitHintSpec

if you want more parallelism. Decreasing the

maxSplitSize

will make more smaller jobs. It’s in the

tuningConfig

section and looks like

"splitHintSpec": {
      "type": "maxSize",
      "maxSplitSize": 2147483648,
      "maxNumFiles": 1000
    }

https://druid.apache.org/docs/latest/ingestion/native-batch.html#split-hint-spec

maxNumConcurrentSubTasks was the winner here. I increased it from 1 to 200 and got the task run time down to half an hour, which is fast enough that I don’t need to try anything else. I did try tweaking splitHintSpec and maxRowsInMemory/maxBytesInMemory but only managed to make it worse in my limited experimentation.