We use confluent Kafka as our Central message bus Is there a DataHub #ingestion

We use confluent Kafka as our Central message bus....

red-pizza-28006

12/02/2021, 3:58 PM

We use confluent Kafka as our Central message bus. Is there anyone who can share an example of their Kafka ingestion, I am not able to get it working, and pretty sure it has to do something with how i am passing the credentials.

Copy code

source:
  type: "kafka"
  config:
    connection:
      bootstrap: "bootstrap:9092"
      consumer_config:
        sasl.username: "youthoughtiwasreal"
        sasl.password: "icantsharethat"
        security.protocol: "sasl_plaintext"
      schema_registry_url: "<https://sr:8081>"
      schema_registry_config:
        <http://basic.auth.user.info|basic.auth.user.info>: nonono:istillcannotshare

sink:
  type: "datahub-rest"
  config:
    server: "<http://gmsendpoint>"

big-carpet-38439

12/02/2021, 4:31 PM

Hey! Can you share the output you get when running ingestion?

red-pizza-28006

12/02/2021, 5:09 PM

Copy code

---- (full traceback above) ----
File "/Users/ajaykumarmuppuri/opt/anaconda3/lib/python3.8/site-packages/datahub/entrypoints.py", line 95, in main
    sys.exit(datahub(standalone_mode=False, **kwargs))
File "/Users/ajaykumarmuppuri/opt/anaconda3/lib/python3.8/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
File "/Users/ajaykumarmuppuri/opt/anaconda3/lib/python3.8/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
File "/Users/ajaykumarmuppuri/opt/anaconda3/lib/python3.8/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/Users/ajaykumarmuppuri/opt/anaconda3/lib/python3.8/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/Users/ajaykumarmuppuri/opt/anaconda3/lib/python3.8/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
File "/Users/ajaykumarmuppuri/opt/anaconda3/lib/python3.8/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
File "/Users/ajaykumarmuppuri/opt/anaconda3/lib/python3.8/site-packages/datahub/cli/ingest_cli.py", line 68, in run
    pipeline = Pipeline.create(pipeline_config, dry_run, preview)
File "/Users/ajaykumarmuppuri/opt/anaconda3/lib/python3.8/site-packages/datahub/ingestion/run/pipeline.py", line 143, in create
    return cls(config, dry_run=dry_run, preview_mode=preview_mode)
File "/Users/ajaykumarmuppuri/opt/anaconda3/lib/python3.8/site-packages/datahub/ingestion/run/pipeline.py", line 109, in __init__
    self.source: Source = source_class.create(
File "/Users/ajaykumarmuppuri/opt/anaconda3/lib/python3.8/site-packages/datahub/ingestion/source/kafka.py", line 76, in create
    return cls(config, ctx)
File "/Users/ajaykumarmuppuri/opt/anaconda3/lib/python3.8/site-packages/datahub/ingestion/source/kafka.py", line 61, in __init__
    self.consumer = confluent_kafka.Consumer(

KafkaException: KafkaError{code=_INVALID_ARG,val=-186,str="Failed to create consumer: Invalid sasl.kerberos.kinit.cmd value: Property not available: "sasl.kerberos.keytab""}

square-activity-64562

12/03/2021, 6:58 AM

Can you look at https://datahubproject.io/docs/deploy/confluent-cloud/ and pass the configuration given here to test out your connectivity? If you see any errors please do share them

red-pizza-28006

12/03/2021, 7:23 AM

I believe this is to use confluent kafka instead of the native kafka datahub uses. I intend to ingest kafka metadata from confluent kafka

square-activity-64562

12/03/2021, 7:50 AM

The issue is with connectivity. At the end of this document are some configurations which are tested for connectivity. I was suggesting to try to use those configurations and see if it solves connectivity in the recipe

square-activity-64562

12/06/2021, 10:01 AM

@red-pizza-28006 Did you manage to solve this?

red-pizza-28006

12/07/2021, 4:55 AM

I will get back to you on this

Open in Slack

Previous Next