Hi! I’m checking schema resolution for kafka topics. According to the code,

TopicName

and

TopicRecordName

strategy are covered (the second in a sort of best-effort mode). However, when testing, we have found that we miss many schemas that follow the

TopicRecordName

strategy. We had a look to the code here https://github.com/datahub-project/datahub/blob/master/metadata-ingestion/src/datahub/ingestion/source/confluent_schema_registry.py#L72-L78

# Subject name format when the schema registry subject name strategy is
        #  (a) TopicNameStrategy(default strategy): <topic name>-<key/value>
        #  (b) TopicRecordNameStrategy: <topic name>-<fully-qualified record name>-<key/value>
        for subject in self.known_schema_registry_subjects:
            if subject.startswith(topic) and subject.endswith(subject_key_suffix):
                return subject
        return None

and we think there is a mistake. The code assumes the

-(key|value)

suffix also exists for

TopicRecordName

strategy, however this is not true; we have plenty of topics with the following format

<topic name>-<fully-qualified record name>

for the

TopicRecordName

strategy. Is there any serializer supporting this

<topic name>-<fully-qualified record name>-<key/value>

format? For example, Confluent AVRO serializer doesn’t https://docs.confluent.io/platform/6.1.2/clients/confluent-kafka-python/html/_modules/confluent_kafka/schema_registry/avro.html As a mitigation, schema resolution may first try exact match for

TopicNameStrategy

and if not, do a search only with

startswith

(no

endswith

). But I think that may bring many false positives and no way to differentiate key and value schemas. So from my understanding, fetching schemas only works for

TopicName

strategy. I’m raising the concern here in case I’m missing something or if someone else has an idea to fix

TopicRecordName

strategy. Thanks! 🧵

Great question... Will wait for others to chime in

@witty-butcher-82399, this seems like it could be an issue closer to the confluent side of things, have you had any luck troubleshooting in that community?

Hi @astonishing-answer-96712 Sorry for the late response, I was on Xmas 🧑‍🎄 holiday I don’t think this is a confluent issue. Instead, it is the DataHub connector the one badly assuming that topics following

TopicRecordNameStrategy

strategy has the

-(key|value)

suffix. So, AFAIK this is not true.

#  (b) TopicRecordNameStrategy: <topic name>-<fully-qualified record name>-<key/value>

Instead it should be:

#  (b) TopicRecordNameStrategy: <topic name>-<fully-qualified record name>

This results in DataHub Kafka connector not fetching schemas for topics following that strategy. I’m just reporting the issue. Additionally, I find it hard to fix since the lack of suffix makes impossible to decide whether found schema (just based on topic name prefix matching) is for the key or for the value.

Ok, thank you sergio, I’ll go ahead and log an issue in git!