question about handling Avro schemas in Flink jobs - we have multiple Kafka topics we're reading from, each containing different Avro messages with different schemas (actually a couple of schema versions per topic). We'd like to run with high parallelism, but when we do we exceed the rate limits on the schema registry (either Confluent Cloud or Glue) at startup as all the parallel instance load their schemas. How do people deal with this issue?
for streaming jobs we've implemented our own try/catch handling with exponential backoff to deal with the registry limits, but for table api jobs there doesn't appear to be any easy way to intercept the exceptions.