I'm trying to configure access to schema registry with basic auth enabled for the GMS. I was able to configure schema registry access for the MAE/MCE services with the following env variables:

- name: SPRING_KAFKA_PROPERTIES_BASIC_AUTH_CREDENTIALS_SOURCE
      value: USER_INFO
    - name: SPRING_KAFKA_PROPERTIES_BASIC_AUTH_USER_INFO
      valueFrom:
        secretKeyRef:
          name: "kafka-schema-registry-credentials"
          key: "user-info"

And the logs from both MAE/MCE look like:

16:21:26.721 [main] INFO  i.c.k.s.KafkaAvroDeserializerConfig - KafkaAvroDeserializerConfig values: 
	schema.registry.url = [redacted]
	<http://basic.auth.user.info|basic.auth.user.info> = [hidden]
	auto.register.schemas = true
	max.schemas.per.subject = 1000
	basic.auth.credentials.source = USER_INFO
	<http://schema.registry.basic.auth.user.info|schema.registry.basic.auth.user.info> = [hidden]
	specific.avro.reader = false
	value.subject.name.strategy = class io.confluent.kafka.serializers.subject.TopicNameStrategy
	key.subject.name.strategy = class io.confluent.kafka.serializers.subject.TopicNameStrategy

16:21:26.857 [main] INFO  o.a.kafka.common.utils.AppInfoParser - Kafka version: 2.2.1-cp1

However, doing the same for the GMS is not working. Specifically I get these warn log messages on startup and the configs are not attached to the serializer:

16:20:23.481 [main] INFO  i.c.k.s.KafkaAvroSerializerConfig - KafkaAvroSerializerConfig values: 
	schema.registry.url = [redacted]
	max.schemas.per.subject = 1000

16:20:24.213 [main] WARN  o.a.k.c.producer.ProducerConfig - The configuration '<http://basic.auth.user.info|basic.auth.user.info>' was supplied but isn't a known config.
16:20:24.215 [main] WARN  o.a.k.c.producer.ProducerConfig - The configuration 'basic.auth.credentials.source' was supplied but isn't a known config.

16:20:24.217 [main] INFO  o.a.kafka.common.utils.AppInfoParser - Kafka version: 2.3.0

When digging into this I noticed the MAE/MCE are using kafka

2.2.1-cp1

(confluent platform version) while the GMS is using

2.3.0

(non-confluent platform version). I'm thinking regular non confluent platform clients might not support the same set of schema registry configurations.

Thanks. Please let us know if that fixes the problem. If not feel free to create an issue on GH and we'll look into further.

That was not it specifically, but it led me down the kafka clients rabbithole and I eventually found out the issue. The MCE/MAE jobs were using newer versions of

kafka-avro-serializer

that supported schema registry configuration while the GMS service was using a very old version that did not.

Thanks for debugging this. I assume this might be the root cause for https://github.com/linkedin/datahub/issues/1861 as well?

Yup that is probably the same issue, we do not use SSL for our schema registry so I did not test those bits but all the other schema registry config was being ignored

Thanks. LGTM. Will merge after CI passes. I assume you've tested this end-to-end?

Yes, I ran

COMPOSE_DOCKER_CLI_BUILD=1 DOCKER_BUILDKIT=1 docker-compose -p datahub build

built the images locally, pushed to dockerhub, deployed them to our k8s cluster, and verified GMS now connects to the schema registry with basic auth and processes events instead of crashing. I tried validating nothing else broke, but I'm still new to datahub so not sure what sort of end to end testing you do.

That's good enough. Unfortunately we don't have automated e2e tests (yet) but it's on the immediately roadmap.

Hi, I did change kafkaAvroSerde to version io.confluentkafka streams avro serde5.5.1, rebuilt app, ran COMPOSE_DOCKER_CLI_BUILD=1 DOCKER_BUILDKIT=1 docker-compose -p datahub build, pushed to dockerhub and deployed to our EKS cluster. When pods restarts I can still see the following in the logs: • The configuration ‘schema.registry.ssl.keystore.password’ was supplied but isn’t a known config. • The configuration ‘schema.registry.ssl.keystore.password’ was supplied but isn’t a known config. • The configuration ‘schema.registry.ssl.truststore.location’ was supplied but isn’t a known config. • The configuration ‘schema.registry.ssl.keystore.location’ was supplied but isn’t a known config. • The configuration ‘schema.registry.ssl.key.password’ was supplied but isn’t a known config. • The configuration ‘schema.registry.ssl.truststore.password’ was supplied but isn’t a known config. So, something missing still….

@silly-apple-97303 ^^^

I unfortunately do not have an SSL enabled schema registry to test with. As @bumpy-keyboard-50565 suggested here I would try using a different configuration key name (and maybe double check with the confluent slack group)