Is the kafka schema registry component strictly required? It seems as though I’m able to successfully ingest from multiple sources (glue, redshift) using the datahub-rest sink without it. I still get tons of logs within datahub-gms related to avro deserialization failing. I’m deploying on kubernetes using the Strimzi operator to define kafka. Strimzi has no official schema registry support, so if I can avoid installing something like the confluent schema registry then I will.

similar issue, interested in the solution to your problem

@dazzling-yak-93039 might be able to speak to this!

Hi, we do use the schema registry for the setup step e.g. creating the elasticsearch indices. This is also used in the upgrade step if the schema changes. So, I think you may not need it for product usage, but it is used during the setup / deployment step.

Got it - thanks for the response! The Helm charts reference

INTERNAL

,

KAFKA

and

AWS_GLUE

registry options - is it possible to select

INTERNAL

or

AWS_GLUE

to specifically avoid the confluent schema registry? What is the difference here?

In the last two releases, there are issues with

INTERNAL

and

AWS_GLUE

options. But, AWS Glue support will be fixed in the next release!

That’s another thing, I’m on chart release

0.2.179

- is that version affected by these issues? datahub-gms version in that chart is

0.2.150

Yes, that version is affected

Gotcha. To clarify - if the

INTERNAL

and

AWS_GLUE

registries could be used, then datahub would not be using the confluent (or other kafka) schema registry component?

That's right!

Thanks! I see that chart version 0.2.186 was released, but I can’t seem to find a changelog. Any chance you could link me to one?

This is the change: https://github.com/acryldata/datahub-helm/pull/366 it is to set the application version in the chart to

0.11.0

, and here are the release notes for this release: https://github.com/datahub-project/datahub/releases

Specifically you may be interested in the "Important Bug Fixes" section: • Glue Schema Registry fixed 🙂

Excellent, thank you!