And the logs from both MAE/MCE look like:
- name: SPRING_KAFKA_PROPERTIES_BASIC_AUTH_CREDENTIALS_SOURCE value: USER_INFO - name: SPRING_KAFKA_PROPERTIES_BASIC_AUTH_USER_INFO valueFrom: secretKeyRef: name: "kafka-schema-registry-credentials" key: "user-info"
However, doing the same for the GMS is not working. Specifically I get these warn log messages on startup and the configs are not attached to the serializer:
16:21:26.721 [main] INFO i.c.k.s.KafkaAvroDeserializerConfig - KafkaAvroDeserializerConfig values: schema.registry.url = [redacted] <http://basic.auth.user.info|basic.auth.user.info> = [hidden] auto.register.schemas = true max.schemas.per.subject = 1000 basic.auth.credentials.source = USER_INFO <http://schema.registry.basic.auth.user.info|schema.registry.basic.auth.user.info> = [hidden] specific.avro.reader = false value.subject.name.strategy = class io.confluent.kafka.serializers.subject.TopicNameStrategy key.subject.name.strategy = class io.confluent.kafka.serializers.subject.TopicNameStrategy 16:21:26.857 [main] INFO o.a.kafka.common.utils.AppInfoParser - Kafka version: 2.2.1-cp1
When digging into this I noticed the MAE/MCE are using kafka
16:20:23.481 [main] INFO i.c.k.s.KafkaAvroSerializerConfig - KafkaAvroSerializerConfig values: schema.registry.url = [redacted] max.schemas.per.subject = 1000 16:20:24.213 [main] WARN o.a.k.c.producer.ProducerConfig - The configuration '<http://basic.auth.user.info|basic.auth.user.info>' was supplied but isn't a known config. 16:20:24.215 [main] WARN o.a.k.c.producer.ProducerConfig - The configuration 'basic.auth.credentials.source' was supplied but isn't a known config. 16:20:24.217 [main] INFO o.a.kafka.common.utils.AppInfoParser - Kafka version: 2.3.0
(confluent platform version) while the GMS is using
(non-confluent platform version). I'm thinking regular non confluent platform clients might not support the same set of schema registry configurations.
and something like
). If this is the correct model, what would be a good way to associate each particular FieldProfile with a particular SchemaField? Or is there a different model that would be better? More generally, it seems like there's a tension between two models for field-level aspects:1. Dataset has many Aspects, some of which have metadata for many Fields 2. Dataset has many Fields, some of which have multiple Aspects I don't have an opinion on which of these models is more "correct", but the current implementation only seems to really support one aspect per field, and pushes any extensions to favour model (1) above. Is this a conscious design decision, or has this question just not come up yet?
I see items in the Upstream/Downstream tables in the “Relationship” tab for all dummy datasets. The lineage data is however empty, as well as the graph visualisation. Am I missing something?
we define e.g. ChartSnapshot and MLModelSnapshot. However, in
we only list MLModelSnapshot and not ChartSnapshot in the union. Why is that? Similary, in
we define a ChartEntity, but not a MLModelEntity, and the ChartEntity is not listed in the union in
. Why is that?