question about handling Avro schemas in Flink jobs...
# troubleshooting
d
question about handling Avro schemas in Flink jobs - we have multiple Kafka topics we're reading from, each containing different Avro messages with different schemas (actually a couple of schema versions per topic). We'd like to run with high parallelism, but when we do we exceed the rate limits on the schema registry (either Confluent Cloud or Glue) at startup as all the parallel instance load their schemas. How do people deal with this issue? for streaming jobs we've implemented our own try/catch handling with exponential backoff to deal with the registry limits, but for table api jobs there doesn't appear to be any easy way to intercept the exceptions.
g
Is self hosting an option?
If not fully host the schema registry maybe some read through cache?
d
we're looking at self hosting, were hoping someone had an easier solution. I also thought about a read-through cache, but this is another piece of infrastructure to be maintained so we may as well go with self hosting the registry.
g
By the way how many request you do on startup and what is the rate limit?
d
the standard rate limit on Confluent and Glue is roughly 40 / second. When we start up our job we're reading several types of records, each with a couple of schemas - so figure about 10 schemas in total within the first second of execution. That's fine for a single instance, but if you're running with reasonable parallelism you're likely to hit the quota, at which point your job dies.