https://datahubproject.io logo
Join Slack
Powered by
hi there im trying to ingest kafka data from confl...
# ingestion
l
hi there im trying to ingest kafka data from confluent cloud... ive got the connection to the kafka broker working but not sure how to pass in the key and token for the schema registry... if i understand correctly... reading through this issue to find out more... https://github.com/linkedin/datahub/issues/1861
g
If I understand correctly, you just want to ingest schema information from schema registry - the referenced issue is about using Kafka/schema registry in datahub’s backend, as datahub is also stream-based.
With the ingestion framework, there’s a good example recipe for connecting to kafka and schema registry securely: https://github.com/linkedin/datahub/blob/master/metadata-ingestion/examples/recipes/secured_kafka_to_console.yml. The options in 
schema_registry_config
are passed directly into the schema registry client: https://docs.confluent.io/platform/current/clients/confluent-kafka-python/html/index.html#confluent_kafka.schema_registry.SchemaRegistryClient
Out of curiosity, what schema languages (e.g. Avro, json schema, protobuf) are you using with schema registry?
l
well those are the three 🙂 let me confirm
g
Also, I saw the github issue that you opened (https://github.com/linkedin/datahub/issues/2504) - I think the 
<http://basic.auth.user.info|basic.auth.user.info>
option is the one you really want for confluent cloud’s schema registry
hey @lively-sunset-25180 - did that config work for you?
l
actually figured out after looking at the confluent kafka python code it accepts the auth in the url...
so yes i got it working 🙂
now i can see the avro schemas in DH
g
huh interesting - did the 
<http://basic.auth.user.info|basic.auth.user.info>
option not work for you, or did you not try it?
in any case, it’s great that you can now see avro schemas in DH 🎉
Open in Slack
PreviousNext
Powered by