Hey everyone, I have a flink job that is reliant o...
# troubleshooting
t
Hey everyone, I have a flink job that is reliant on an avro model and schema, my job is also backed by rocksdb. Whenever I update the schema (fully backwards compatible change) the process fails when trying to load the objects from rocksdb and i have to delete the state and start fresh. Is there a way to properly load the state when updating the avro model?
p
Hello there Tomer! Do what are the values of your configs: • pipeline.generic-types • pipeline.auto-generate-uids
t
Hi Pedro, thanks for the reply, I actually do not set these
👋 1
p
Perhaps this may be the issue. The first one, when set to false, will ensure you do not use Kryo serialization, which may cause schema evolution issues. The second one, when set to false, will require you to set uid for each operator and it will ensure when flink is recovering state things are still mappable. I hope this helps with your current issue.
t
Thanks Pedro, no Kryo, just pure Avro
p
AVRO is the schema. But flink may be using kryo to handle the objects. A faster way to test it is setting the property to false and seeing if the pipeline runs. It shows a runtime error when loading the pipeline.
t
well avro is also the objects, the avro serialization uses the avro datum to serialize/deserialize
p
Gotcha. I'd enable it anyway to be sure together with the uid. If you need help lmk I can spare some 15" in a call 😁
k
I ran into a similar issue created - https://issues.apache.org/jira/browse/FLINK-38023