This message was deleted.

I found this article that explains how to do this using Overload Dynamic Config -> Select strategy and an "equalDistributionWithCategorySpec"

Hi @Sergio Ferragut Thanks for the article. We have already deployed similar configs

{
  "type": "equalDistributionWithCategorySpec",
  "workerCategorySpec": {
    "categoryMap": {
      "single_phase_sub_task": {
        "defaultCategory": "subtask_category",
        "categoryAffinity": {}
      },
      "index": {
        --
      },
      "index_parallel": {
        --
      },
      "compact": {
        --
      },
      "kill": {
        --
      }
    },
    "strong": true
  }
}

however now we need to distinguish between

single_phase_sub_task

from compaction task and index_parallel_task. is there any way to do that?

Why not dedicate compaction into its own mm tier?

ok if i dedicate compact to own tier will

single_phase_sub_task

from compaction task will go to that tier

yes

ok i will test that and let you know

to my knowledge compaction is a simple long running process. It doesn’t spawn single_phase_sub_task

yes for us it spins subtasks as it has huge data

Putting compaction inside its own tier is how we do it on all of our clusters because its disk and memory requirements are very different than ingestion.

i am trying to use below config but some how jobs are in pending state for the second datasource . we are using Druid 25 and ds2 datasource we are using different set of MM but when we submit a job it is in pending state and not getting assigned to any MM .

{
  "type": "equalDistributionWithCategorySpec",
  "workerCategorySpec": {
    "categoryMap": {
      "index_parallel": {
        "defaultCategory": "_default_worker_category",
        "categoryAffinity": {
          "ds1": "ds1_batch",
          "ds2": "ds2_batch"
        }
      },
      "partial_dimension_cardinality": {
        "defaultCategory": "_default_worker_category",
        "categoryAffinity": {
          "ds1": "ds1_batch",
          "ds2": "ds2_batch"
        }
      },
      "partial_index_generate": {
        "defaultCategory": "_default_worker_category",
        "categoryAffinity": {
          "ds1": "ds1_batch",
          "ds2": "ds2_batch"
        }
      },
      "partial_index_generic_merge": {
        "defaultCategory": "_default_worker_category",
        "categoryAffinity": {
          "ds1": "ds1_batch",
          "ds2_perf": "ds2_batch"
        }
      },
      "partial_range_index_generate": {
        "defaultCategory": "_default_worker_category",
        "categoryAffinity": {
          "ds1": "ds1_batch"
        }
      },
      "partial_dimension_distribution": {
        "defaultCategory": "_default_worker_category",
        "categoryAffinity": {
          "ds1": "ds1_batch"
        }
      }
    },
    "strong": true
  }
}