Just curious, is it possible to run datahub without confluent-schema-registry?

Not currently, we are looking into removing it as a requirement though: https://feature-requests.datahubproject.io/b/Developer-Experience/p/remove-required-dependency-on-confluent-schema-registry

Do you know the rough ETA? 1year? 3year?

Definitely not that lengthy of a time horizon 🙂 We don't have someone actively working on it right at this moment, but this is a priority for us within the next few months. @little-megabyte-1074 can keep me honest when her flight lands. Note: If your issue is just with Confluent's schema registry, there are alternatives like Karapace that you can use to get around it in the immediate.

Thanks for the context. I want to remove it because we are required to deploy using the image baked internally. And I realized cp-schema-registry image is difficult to build, it requires lots of other dependencies and we need to set up internal repo for cp-schema-registry.

@mysterious-lamp-91034: what schema registry do you use internally at Pinterest?

We don't have schema registry😅 I just run cp-schema-registry image in the same host with datahub

Oh! so what about the Kafka topics that Pinterest runs internally? Are those schema-less?

Hi friends! @mysterious-lamp-91034 it's definitely on our near-term radar; we’re firming up Q2 plans by the end of the month - it will either by Q2 or Q3 at the latest

@mammoth-bear-12532 I am new hire I am not sure 😅 But I am sure schema-registry are not used in Pinterest and the content in some topics are thrift based.

Hi @eager-lifeguard-22029! It’s still a required dependency, but we do have a community member that has a WIP PR open here. I just sent him a message to see if we have any updates on next steps/timeline

Thanks @little-megabyte-1074!

There has been at least one successful setup with Karapace as well, just as an aside

Any idea if it's possible to use Azure Schema Registry in Azure Events Hubs as an alternative?

Don't think anyone has tried, if it doesn't have drop-in compatibility with Confluent Schema Registry there are probably code changes required though. Based on those docs it looks like it uses

KafkaAvroDeserializer

so it might be possible if that's the class from the Confluent library and not a custom class of the same name.