Jérôme Gagnon
10/11/2024, 3:35 PMConfluentRegistryAvroSerializationSchema ? I saw a stackoverflow from @Martijn Visser recommending to use auto.register.schemas and use.latest.version but I cannot get it to work in any possible way.D. Draco O'Brien
10/11/2024, 5:04 PMauto.register.schemas and use.latest.version, it seems you’re on the right track, but there might be some nuances to consider.
Here’s a general guide on how you can achieve this:
1. Set auto.register.schemas to false: This configuration tells the schema registry client not to automatically register new schemas.
2. Avoid using use.latest.version unless necessary: This property is typically used to indicate whether the serializer should use the latest version of a schema when serializing data. If you’re disabling auto-registration, you likely have pre-registered schemas and would want to reference them explicitly, so this might not be needed or could be misleading if misunderstood.
To apply these configurations, you’d typically pass them when initializing your Kafka producer or serializer like so (assuming you’re using Java and Kafka Clients):
Properties props = new Properties();
props.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, "your.bootstrap.servers");
// Other necessary producer configs...
// Disable auto schema registration
props.put(AbstractKafkaSchemaSerDeConfig.AUTO_REGISTER_SCHEMAS, false);
// Optionally, specify the specific schema to use (if not using the latest by default)
// props.put(KafkaAvroSerializerConfig.SCHEMA_REGISTRY_URL_CONFIG, "http://schema.registry.url");
// props.put(KafkaAvroSerializerConfig.VALUE_SUBJECT_NAME_STRATEGY, TopicNameStrategy.class.getName()); // Or other strategy as needed
// props.put(KafkaAvroSerializerConfig.SCHEMA_ID_CONFIG, "<schema-id>"); // Only if you're not using subject naming strategy
KafkaProducer<String, YourAvroType> producer = new KafkaProducer<>(props,
new StringSerializer(),
new KafkaAvroSerializer(schemaRegistryClient)); // Ensure your serializer has the correct config
// Then proceed to send records...
A few important points:
• Make sure your KafkaAvroSerializer instance (or whichever Avro Serializer you’re using that interacts with the Schema Registry) is configured with the correct settings, including the URL to your Schema Registry and potentially any custom subject naming strategies if you’re not directly specifying schema IDs.
• If you’re using Spring Boot or other frameworks with their own abstractions, the way to configure these properties might differ slightly. Look for equivalent configuration keys or methods to set these properties in your framework of choice.
• When disabling auto-registration, ensure all schemas you intend to use are pre-registered manually through the Schema Registry’s REST API or UI to avoid runtime exceptions due to missing schemas.
If you’ve tried these steps and are still encountering issues, it might help to review the exact error messages so please share it with us.Jérôme Gagnon
10/11/2024, 5:56 PM2024-10-11 13:52:02
java.io.IOException: Could not register schema in registry
at org.apache.flink.formats.avro.registry.confluent.ConfluentSchemaRegistryCoder.writeSchema(ConfluentSchemaRegistryCoder.java:90)
at org.apache.flink.formats.avro.RegistryAvroSerializationSchema.serialize(RegistryAvroSerializationSchema.java:120)
at org.apache.flink.connector.kafka.sink.KafkaRecordSerializationSchemaBuilder$KafkaRecordSerializationSchemaWrapper.serialize(KafkaRecordSerializationSchemaBuilder.java:334)
at org.apache.flink.connector.kafka.sink.KafkaWriter.write(KafkaWriter.java:193)
at org.apache.flink.streaming.runtime.operators.sink.SinkWriterOperator.processElement(SinkWriterOperator.java:160)
at org.apache.flink.streaming.runtime.tasks.CopyingChainingOutput.pushToOperator(CopyingChainingOutput.java:75)
at org.apache.flink.streaming.runtime.tasks.CopyingChainingOutput.collect(CopyingChainingOutput.java:50)
at org.apache.flink.streaming.runtime.tasks.CopyingChainingOutput.collect(CopyingChainingOutput.java:29)
at org.apache.flink.streaming.api.operators.StreamMap.processElement(StreamMap.java:38)
at org.apache.flink.streaming.runtime.io.RecordProcessorUtils.lambda$getRecordProcessor$0(RecordProcessorUtils.java:60)
at org.apache.flink.streaming.runtime.tasks.OneInputStreamTask$StreamTaskNetworkOutput.emitRecord(OneInputStreamTask.java:238)
at org.apache.flink.streaming.runtime.io.AbstractStreamTaskNetworkInput.processElement(AbstractStreamTaskNetworkInput.java:157)
at org.apache.flink.streaming.runtime.io.AbstractStreamTaskNetworkInput.emitNext(AbstractStreamTaskNetworkInput.java:114)
at org.apache.flink.streaming.runtime.io.StreamOneInputProcessor.processInput(StreamOneInputProcessor.java:65)
at org.apache.flink.streaming.runtime.tasks.StreamTask.processInput(StreamTask.java:579)
at org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.runMailboxLoop(MailboxProcessor.java:231)
at org.apache.flink.streaming.runtime.tasks.StreamTask.runMailboxLoop(StreamTask.java:909)
at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:858)
at org.apache.flink.runtime.taskmanager.Task.runWithSystemExitMonitoring(Task.java:958)
at org.apache.flink.runtime.taskmanager.Task.restoreAndInvoke(Task.java:937)
at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:751)
at org.apache.flink.runtime.taskmanager.Task.run(Task.java:566)
at java.base/java.lang.Thread.run(Unknown Source)
Caused by: io.confluent.kafka.schemaregistry.client.rest.exceptions.RestClientException: User is denied operation Write on Subject: [REDACTED]; error code: 40301
at io.confluent.kafka.schemaregistry.client.rest.RestService.sendHttpRequest(RestService.java:301)
at io.confluent.kafka.schemaregistry.client.rest.RestService.httpRequest(RestService.java:371)
at io.confluent.kafka.schemaregistry.client.rest.RestService.registerSchema(RestService.java:548)
at io.confluent.kafka.schemaregistry.client.rest.RestService.registerSchema(RestService.java:536)
at io.confluent.kafka.schemaregistry.client.rest.RestService.registerSchema(RestService.java:494)
at io.confluent.kafka.schemaregistry.client.CachedSchemaRegistryClient.registerAndGetId(CachedSchemaRegistryClient.java:274)
at io.confluent.kafka.schemaregistry.client.CachedSchemaRegistryClient.register(CachedSchemaRegistryClient.java:381)
at io.confluent.kafka.schemaregistry.client.CachedSchemaRegistryClient.register(CachedSchemaRegistryClient.java:360)
at io.confluent.kafka.schemaregistry.client.CachedSchemaRegistryClient.register(CachedSchemaRegistryClient.java:348)
at io.confluent.kafka.schemaregistry.client.SchemaRegistryClient.register(SchemaRegistryClient.java:47)
at org.apache.flink.formats.avro.registry.confluent.ConfluentSchemaRegistryCoder.writeSchema(ConfluentSchemaRegistryCoder.java:85)
... 22 moreJérôme Gagnon
10/11/2024, 8:43 PMD. Draco O'Brien
10/12/2024, 7:10 AMD. Draco O'Brien
10/12/2024, 7:14 AMJérôme Gagnon
10/12/2024, 1:56 PMD. Draco O'Brien
10/12/2024, 4:08 PMD. Draco O'Brien
10/12/2024, 4:09 PMD. Draco O'Brien
10/12/2024, 4:17 PMD. Draco O'Brien
10/12/2024, 4:17 PMD. Draco O'Brien
10/12/2024, 4:19 PMD. Draco O'Brien
10/12/2024, 4:19 PMD. Draco O'Brien
10/12/2024, 4:20 PMD. Draco O'Brien
10/12/2024, 4:21 PMJérôme Gagnon
10/15/2024, 2:46 PMAlso make sure if there all jar files are correct for your version of FlinkThese are all correct
Jérôme Gagnon
10/15/2024, 3:03 PMLooking around I dont see any open flink issues that match. (edited)I didn’t either
Jérôme Gagnon
11/01/2024, 1:27 PM