This message was deleted.

If you can set environment variable in your MiddleManager nodes, you can use Dynamic Config Providers to set the rack in your supervisor like:

{
  "type": "kafka",
  "spec": {
    "ioConfig": {
      "type": "kafka",
      "consumerProperties": {
        "bootstrap.servers": "ignoreServer",
        "druid.dynamic.config.provider": {
          "type": "environment",
          "variables": {
            "bootstrap.servers": "KAFKA_SERVER",
            "client.rack": "CONSUMER_CLIENT_RACK"
          }
        }
      },
      "topic": "thingy"
    },
    "tuningConfig": {
      "type": "kafka"
    }
  }
}

Here

KAFKA_SERVER

and

CONSUMER_CLIENT_RACK

are environment variables. @Saydul Bashar - anything to add as I gained this knowledge from you 😉

Thank you @Kyle Hoondert. The core elements are already covered in your comment. One more thing to add here is that the client rack environment variable needs to be available in the process that runs the ingestion. I usually add it in the script that starts the druid process itself. I am currently working on a blog post for this that should come out in the next few weeks.

Thanks @Kyle Hoondert. Hi @Saydul Bashar , I didn't understand your statement.

client rack environment variable needs to be available in the process that runs the ingestion. I usually add it in the script that starts the druid process itself.

Please enlighten me on this. I'm new to Druid Kafka Ingestion.

Hi @Vineeth thank you for reaching out. As you can see from the dynamic config provided by @Kyle Hoondert the client rack information is

"client.rack": "CONSUMER_CLIENT_RACK"

. Here :

CONSUMER_CLIENT_RACK

is an environment variable that you need to set on the process that runs the kafka consumer. In my case I wrote a user init script that runs when every node starts and added this there:

#!/bin/bash

# And add the environment variable before starting the Imply processes
sed -i '/^exec.*/i export\ CONSUMER_CLIENT_RACK=`curl -s <http://169.254.169.254/latest/meta-data/placement/availability-zone>` ' $(find /opt/grove -name run-druid | grep -v dist)

This works for druid setups that are running on linux and gets initialised by a run-druid setup. For druid setups that use something else you will need to find the script that actually starts the druid processes and add the environment variable before the process initialisation. @Sergio Ferragut FYI this is related to the discussion we were having yesterday. So, I thought you may be interested.

Thanks @Saydul Bashar, I'll try this.

You are welcome 🙂