Hi Team, I have ingested data from kafka but I am ...
# troubleshoot
n
Hi Team, I have ingested data from kafka but I am not able to see any schema. Every thing is null. I have around 70 datasets, all are blank Can someone help me
w
I'm not very up to speed with the kafka source but before at least, it required that the schema for a given topic followed a certain naming pattern for it to be found.
n
@wonderful-quill-11255 I didn't get it. Can you elaborate a bit more?
w
One moment, let me find the code and link it to you
So basically, if you have a topic called "my-topic", the schema will be looked up in the schema registry by the key "my-topic-key" for the key-schema, and "my-topic-value" for the value-schema.
At my company, less than half of the topic owners have adhered to this convention so we have the same problem you have with no schemas in the catalog for those topics.
It looks like you can override it in the kafka source config though, on a per topic level.
n
how to override it?
w
There is also an example a bit further down.
n
But do I have to provide custom schema for all 70 tables manually?
w
If the schemas are following some other convention you can probably follow this guide. https://datahubproject.io/docs/generated/ingestion/sources/kafka#custom-schema-registry If your topics follow no convention at all (this is my situation, and we have a lot more than 70 topics ...), then yes, you would have to declare them all.
Note, I'm not working for Acryl/Datahub so take my replies for best effort answers.
n
I am still confused that how did you declare the schema for the topics
I have read the doc
but not able to understand the following topic_subject_map: _# Defines both key & value schema for topic 'my_topic_1'_ "my_topic_1-key": "io.acryl.Schema1" "my_topic_1-value": "io.acryl.Schema2" _# Defines only the value schema for topic 'my_topic_2' (the topic doesn't have a key schema)._ "my_topic_2-value": "io.acryl.Schema3"
w
Are you familiar with the schema-registry component of the kafka eco-system. If not this is a good starting point: https://docs.confluent.io/platform/current/schema-registry/index.html
a
Hey @numerous-account-62719, are you still having trouble here?
n
Hi @astonishing-answer-96712 Yes still cant see the schema
a
Have you read up on the Kafka documentation to better understand topics? This seems like a misconfiguration issue