Hi Team, How can I run a DataHub Actions Framework in the DataHub running on Kubernetes (installed with helm)? I want to run a Custom Action that I'm able to run with DataHub on my localhost (with quickstart). For testing, I tried updating the URL in the connection of hello_world.yaml with the IP address of DataHub but it failed with below error. (Obfuscated the IP below)

Failed to instantiate Actions Pipeline using config {'name': 'hello_world', 'source': {'type': 'kafka', 'config': {'connection': {'bootstrap': '<http://34.1XX.XX.XXX:9092>', 'schema_registry_url': '<http://34.1XX.XX.XXX:8081>'}}}, 'action': {'type': 'hello_world'}} due to
                'Caught exception while attempting to instantiate Event Source of type kafka' due to
                        '1 validation error for KafkaEventSourceConfig
connection -> bootstrap
  host contains bad characters, found <http://34.1XX.XX.XXX> (type=assertion_error)'.
        Run with --debug to get full stacktrace.
        e.g. 'datahub --debug actions -c hello_world.yaml'

@early-airplane-84388 it seems like you are using

http://...

in your kafka bootstrap config ... when it is usually just the raw ip address or hostname of your kafka broker

Also tried that but wasn't able to setup the connection

%4|1663562981.536|FAIL|rdkafka#consumer-1| [thrd:<http://34.1XX.XX.XXX:9092/bootstrap|34.1XX.XX.XXX:9092/bootstrap>]: <http://34.1XX.XX.XXX:9092/bootstrap|34.1XX.XX.XXX:9092/bootstrap>: Connection setup timed out in state CONNECT (after 30031ms in state CONNECT)
%4|1663563012.535|FAIL|rdkafka#consumer-1| [thrd:<http://34.1XX.XX.XXX:9092/bootstrap|34.1XX.XX.XXX:9092/bootstrap>]: <http://34.1XX.XX.XXX:9092/bootstrap|34.1XX.XX.XXX:9092/bootstrap>: Connection setup timed out in state CONNECT (after 30031ms in state CONNECT), 1 identical error(s) suppressed)
...

Here is the hello_word.yaml example that I'm using:

name: "hello_world"
source:
  type: "kafka"
  config:
    connection:
      bootstrap: ${KAFKA_BOOTSTRAP_SERVER:-<http://34.1XX.XX.XXX:9092|34.1XX.XX.XXX:9092>}
      schema_registry_url: ${SCHEMA_REGISTRY_URL:-<http://34.1XX.XX.XXX:8081>}
action:
  type: "hello_world"

could you use an off-the-shelf kafka consumer client (like kcat) to connect to kafka and make sure you have the coordinates right?

Tried connecting with kcat

kcat -b <http://34.1XX.XX.XXX:9092|34.1XX.XX.XXX:9092> -X receive.message.max.bytes=2147483647 -t EntityChangeEvent_v1

But got the below error ->

% ERROR: Failed to query metadata for topic EntityChangeEvent_v1: Local: Broker transport failure

PS: In localhost I'm able to run the DataHub Action & also connect via kcat but couldn't do so for the DataHub on Kubernetes.

That's probably it!

Hello Ajay, Did you deploy your Kafka instance in K8s or are you using a managed service?

Hey @incalculable-ocean-74010, I've deployed it in K8s (GKE cluster) using helm charts to install DataHub and its prerequisites. I'm currently trying to run hello_world.yaml for testing but need to run a custom action with DataHub on K8s. Please suggest the best way to do so.

Can you share the command you ran to deploy the pre-requisite?