Has anyone managed to disable Avro schema registry...
# troubleshooting
j
Has anyone managed to disable Avro schema registry to auto-register the schema with
ConfluentRegistryAvroSerializationSchema
? I saw a stackoverflow from @Martijn Visser recommending to use
auto.register.schemas
and
use.latest.version
but I cannot get it to work in any possible way.
d
Disabling the auto-registration of schemas in Confluent’s Schema Registry when using `ConfluentRegistryAvroSerializationSchema`involves configuring the properties correctly when setting up your producer or serializer. Based on your mention of
auto.register.schemas
and
use.latest.version
, it seems you’re on the right track, but there might be some nuances to consider. Here’s a general guide on how you can achieve this: 1. Set
auto.register.schemas
to
false
: This configuration tells the schema registry client not to automatically register new schemas. 2. Avoid using
use.latest.version
unless necessary
: This property is typically used to indicate whether the serializer should use the latest version of a schema when serializing data. If you’re disabling auto-registration, you likely have pre-registered schemas and would want to reference them explicitly, so this might not be needed or could be misleading if misunderstood. To apply these configurations, you’d typically pass them when initializing your Kafka producer or serializer like so (assuming you’re using Java and Kafka Clients):
Copy code
Properties props = new Properties();
props.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, "your.bootstrap.servers");
// Other necessary producer configs...

// Disable auto schema registration
props.put(AbstractKafkaSchemaSerDeConfig.AUTO_REGISTER_SCHEMAS, false);

// Optionally, specify the specific schema to use (if not using the latest by default)
// props.put(KafkaAvroSerializerConfig.SCHEMA_REGISTRY_URL_CONFIG, "http://schema.registry.url");
// props.put(KafkaAvroSerializerConfig.VALUE_SUBJECT_NAME_STRATEGY, TopicNameStrategy.class.getName()); // Or other strategy as needed
// props.put(KafkaAvroSerializerConfig.SCHEMA_ID_CONFIG, "<schema-id>"); // Only if you're not using subject naming strategy

KafkaProducer<String, YourAvroType> producer = new KafkaProducer<>(props,
    new StringSerializer(),
    new KafkaAvroSerializer(schemaRegistryClient)); // Ensure your serializer has the correct config

// Then proceed to send records...
A few important points: • Make sure your
KafkaAvroSerializer
instance (or whichever Avro Serializer you’re using that interacts with the Schema Registry) is configured with the correct settings, including the URL to your Schema Registry and potentially any custom subject naming strategies if you’re not directly specifying schema IDs. • If you’re using Spring Boot or other frameworks with their own abstractions, the way to configure these properties might differ slightly. Look for equivalent configuration keys or methods to set these properties in your framework of choice. • When disabling auto-registration, ensure all schemas you intend to use are pre-registered manually through the Schema Registry’s REST API or UI to avoid runtime exceptions due to missing schemas. If you’ve tried these steps and are still encountering issues, it might help to review the exact error messages so please share it with us.
j
Thank you for your detailed answer; I have configured what you’re recommending here; however we are still seeing the Serializer trying to call the registry - and since we have ACLs in place to prevent “Write” operation on subject we get the following exception :
Copy code
2024-10-11 13:52:02
java.io.IOException: Could not register schema in registry
	at org.apache.flink.formats.avro.registry.confluent.ConfluentSchemaRegistryCoder.writeSchema(ConfluentSchemaRegistryCoder.java:90)
	at org.apache.flink.formats.avro.RegistryAvroSerializationSchema.serialize(RegistryAvroSerializationSchema.java:120)
	at org.apache.flink.connector.kafka.sink.KafkaRecordSerializationSchemaBuilder$KafkaRecordSerializationSchemaWrapper.serialize(KafkaRecordSerializationSchemaBuilder.java:334)
	at org.apache.flink.connector.kafka.sink.KafkaWriter.write(KafkaWriter.java:193)
	at org.apache.flink.streaming.runtime.operators.sink.SinkWriterOperator.processElement(SinkWriterOperator.java:160)
	at org.apache.flink.streaming.runtime.tasks.CopyingChainingOutput.pushToOperator(CopyingChainingOutput.java:75)
	at org.apache.flink.streaming.runtime.tasks.CopyingChainingOutput.collect(CopyingChainingOutput.java:50)
	at org.apache.flink.streaming.runtime.tasks.CopyingChainingOutput.collect(CopyingChainingOutput.java:29)
	at org.apache.flink.streaming.api.operators.StreamMap.processElement(StreamMap.java:38)
	at org.apache.flink.streaming.runtime.io.RecordProcessorUtils.lambda$getRecordProcessor$0(RecordProcessorUtils.java:60)
	at org.apache.flink.streaming.runtime.tasks.OneInputStreamTask$StreamTaskNetworkOutput.emitRecord(OneInputStreamTask.java:238)
	at org.apache.flink.streaming.runtime.io.AbstractStreamTaskNetworkInput.processElement(AbstractStreamTaskNetworkInput.java:157)
	at org.apache.flink.streaming.runtime.io.AbstractStreamTaskNetworkInput.emitNext(AbstractStreamTaskNetworkInput.java:114)
	at org.apache.flink.streaming.runtime.io.StreamOneInputProcessor.processInput(StreamOneInputProcessor.java:65)
	at org.apache.flink.streaming.runtime.tasks.StreamTask.processInput(StreamTask.java:579)
	at org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.runMailboxLoop(MailboxProcessor.java:231)
	at org.apache.flink.streaming.runtime.tasks.StreamTask.runMailboxLoop(StreamTask.java:909)
	at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:858)
	at org.apache.flink.runtime.taskmanager.Task.runWithSystemExitMonitoring(Task.java:958)
	at org.apache.flink.runtime.taskmanager.Task.restoreAndInvoke(Task.java:937)
	at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:751)
	at org.apache.flink.runtime.taskmanager.Task.run(Task.java:566)
	at java.base/java.lang.Thread.run(Unknown Source)
Caused by: io.confluent.kafka.schemaregistry.client.rest.exceptions.RestClientException: User is denied operation Write on Subject: [REDACTED]; error code: 40301
	at io.confluent.kafka.schemaregistry.client.rest.RestService.sendHttpRequest(RestService.java:301)
	at io.confluent.kafka.schemaregistry.client.rest.RestService.httpRequest(RestService.java:371)
	at io.confluent.kafka.schemaregistry.client.rest.RestService.registerSchema(RestService.java:548)
	at io.confluent.kafka.schemaregistry.client.rest.RestService.registerSchema(RestService.java:536)
	at io.confluent.kafka.schemaregistry.client.rest.RestService.registerSchema(RestService.java:494)
	at io.confluent.kafka.schemaregistry.client.CachedSchemaRegistryClient.registerAndGetId(CachedSchemaRegistryClient.java:274)
	at io.confluent.kafka.schemaregistry.client.CachedSchemaRegistryClient.register(CachedSchemaRegistryClient.java:381)
	at io.confluent.kafka.schemaregistry.client.CachedSchemaRegistryClient.register(CachedSchemaRegistryClient.java:360)
	at io.confluent.kafka.schemaregistry.client.CachedSchemaRegistryClient.register(CachedSchemaRegistryClient.java:348)
	at io.confluent.kafka.schemaregistry.client.SchemaRegistryClient.register(SchemaRegistryClient.java:47)
	at org.apache.flink.formats.avro.registry.confluent.ConfluentSchemaRegistryCoder.writeSchema(ConfluentSchemaRegistryCoder.java:85)
	... 22 more
d
What version of Flink are you using?
It looks like its always trying to register even when auto-register is disabled. I would first check if newer versions or patches fix this issue. If that’s not an option you are probably looking at needing to create a custom serialization schema that extends RegistryAvroSerializationSchema or ConfluentSchemaRegistryCoder depending on where the fix needs to be applied.
j
1.19 - but I don’t believe it changed in 1.20 (latest)
d
ok, so are you interested in doing custom implementation of the registry process?
check for issues first I guess
this same error appeared here. User denied access I think is key.
Not sure these are related but you might want to have a look.
Also make sure if there all jar files are correct for your version of Flink
Looking around I dont see any open flink issues that match.
j
Also make sure if there all jar files are correct for your version of Flink
These are all correct
Looking around I dont see any open flink issues that match. (edited)
I didn’t either
I ended up creating the following issue : https://issues.apache.org/jira/browse/FLINK-36650
✅ 1