Hi team! I'm testing DataHub Actions and was able to complete a custom action I wrote when teh DataHub server and the action are run in the same AWS instance. However, when I try to run them in separate instances, I cannot reach the Kafka Bootsrap Server due to this error:

[2024-01-16 21:35:50,652] INFO     {datahub_actions.cli.actions:76} - DataHub Actions version: 0.0.14
%3|1705440950.713|FAIL|rdkafka#consumer-1| [thrd:GroupCoordinator]: GroupCoordinator: localhost:9092: Connect to ipv4#127.0.0.1:9092 failed: Connection refused (after 0ms in state CONNECT)

The config for the action looks like this:

source:
  type: "kafka"
  config:
    connection:
      bootstrap: "private-ip-datahub-instance:9092"
      schema_registry_url: "<http://private-ip-datahub-instance:8081>"

Why does it try to connect to

localhost

if I'm specifying an IP address? Thank you in advance for your suggestions!

that is very very strange - it should not be using localhost if you are providing this in the configuration, i've never seen this issue before. The private ip is something like 10.0.1.x right?

Hi @big-carpet-38439 , thank you for your reply! Yes, the private IP address is an IPv4 one. I'm deploying it using datahub actions command and the config yaml file I created for a custom action:

datahub actions -c datahub_event.yaml

I see this in the broker environmemt variable: KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://broker:29092,PLAINTEXT_HOST://localhost:9092 @brainy-tent-14503 is the environment variable

@big-carpet-38439 @brainy-tent-14503 I confirm that the above change and exposing the port resolved the issue. Thank you so much for your help! 🙂

🙌🔥