im trying to add a field to my state but when i ru...
# troubleshooting
m
im trying to add a field to my state but when i run my job with new version and pass the last savepoint path with Allow Non Restored State, i get this error com.esotericsoftware.kryo.KryoException: java.lang.IndexOutOfBoundsException: Index 50 out of bounds for length 2 Serialization trace: productId (models.Quantity) at com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:125) at com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:528) at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:761) at org.apache.flink.api.java.typeutils.runtime.kryo.KryoSerializer.deserialize(KryoSerializer.java:394) at org.apache.flink.contrib.streaming.state.RocksDBValueState.value(RocksDBValueState.java:88) at org.apache.flink.runtime.state.metrics.LatencyTrackingValueState.lambda$value$0(LatencyTrackingValueState.java:60) at org.apache.flink.runtime.state.metrics.AbstractLatencyTrackState.trackLatencyWithIOException(AbstractLatencyTrackState.java:112) at org.apache.flink.runtime.state.metrics.LatencyTrackingValueState.value(LatencyTrackingValueState.java:59) at ir.okala.flink.operators.QuantityPriceAggregatorFunction.processElement1(QuantityPriceAggregatorFunction.java:49) at ir.okala.flink.operators.QuantityPriceAggregatorFunction.processElement1(QuantityPriceAggregatorFunction.java:19) at org.apache.flink.streaming.api.operators.co.KeyedCoProcessOperator.processElement1(KeyedCoProcessOperator.java:78) at org.apache.flink.streaming.runtime.io.RecordProcessorUtils.lambda$getRecordProcessor1$1(RecordProcessorUtils.java:95) at org.apache.flink.streaming.runtime.io.StreamTwoInputProcessorFactory$StreamTaskNetworkOutput.emitRecord(StreamTwoInputProcessorFactory.java:254) at org.apache.flink.streaming.runtime.io.AbstractStreamTaskNetworkInput.processElement(AbstractStreamTaskNetworkInput.java:157) at org.apache.flink.streaming.runtime.io.AbstractStreamTaskNetworkInput.emitNext(AbstractStreamTaskNetworkInput.java:114) at org.apache.flink.streaming.runtime.io.StreamOneInputProcessor.processInput(StreamOneInputProcessor.java:65) at org.apache.flink.streaming.runtime.io.StreamMultipleInputProcessor.processInput(StreamMultipleInputProcessor.java:85) at org.apache.flink.streaming.runtime.tasks.StreamTask.processInput(StreamTask.java:638) at org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.runMailboxLoop(MailboxProcessor.java:231) at org.apache.flink.streaming.runtime.tasks.StreamTask.runMailboxLoop(StreamTask.java:973) at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:917) at org.apache.flink.runtime.taskmanager.Task.runWithSystemExitMonitoring(Task.java:970) at org.apache.flink.runtime.taskmanager.Task.restoreAndInvoke(Task.java:949) at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:763) at org.apache.flink.runtime.taskmanager.Task.run(Task.java:575) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.lang.IndexOutOfBoundsException: Index 50 out of bounds for length 2 at java.base/jdk.internal.util.Preconditions.outOfBounds(Unknown Source) at java.base/jdk.internal.util.Preconditions.outOfBoundsCheckIndex(Unknown Source) at java.base/jdk.internal.util.Preconditions.checkIndex(Unknown Source) at java.base/java.util.Objects.checkIndex(Unknown Source) at java.base/java.util.ArrayList.get(Unknown Source) at com.esotericsoftware.kryo.util.MapReferenceResolver.getReadObject(MapReferenceResolver.java:42) at com.esotericsoftware.kryo.Kryo.readReferenceOrNull(Kryo.java:805) at com.esotericsoftware.kryo.Kryo.readObjectOrNull(Kryo.java:728) at com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:113) ... 25 more productid is a string what should i do?
a
I guess it depends on whether you need the state or not? If you don't need the state, then it there are several ways do drop it entirely, or parts of it. If you want to partially drop the state, then you could change the UID of the
QuantityPriceAggregatorFunction
operation, and any other stateful operations that use the type in question. That would effectively mean those stateful operators start from zero and all state for them is lost. So be careful, and ensure that you really don't need that state! Kryo serialization doesn't allow state schema evolution. If you want to be able to add and remove fields you need to use another data type, e.g. POJO or Avro, or use custom state serialization. I have never myself needed to evolve a Kryo serialized type, so I have very little guidance to offer on how to retain the state and evolve to Kryo serialized type. Perhaps the docs on custom serialization helps? We have disabled Kryo serialization entirely by default in all our Flink jobs, to avoid these kinds of issues. This forces the engineers to do some more work in order to ensure all types are serializable using the POJO serializer, but it keeps us out of these kinds of issues. How get migrate from Kryo to POJO is probably not simple, and I cannot provide any guidance on that either (I imagine it would be painful if you want to retain the state)
p
@Andreas Bube has a really complete response. A good tip may be disabling Kryo for new pipelines alongside with disabling auto generated uid for operators
m
thanks a lot @Andreas Bube @Pedro Mázala Fortunately, I can rerun the job from the beginning again, and keeping the state is no longer important to me right now. But I want to make sure this doesn’t happen again in the future.
Copy code
env.getConfig().registerPojoType(Quantity.class);
env.getConfig().registerPojoType(Product.class);
env.getConfig().registerPojoType(CatalogItem.class);
env.getConfig().registerPojoType(Price.class);
env.getConfig().disableForceKryo();
I added these lines of code. Is this the correct approach? And if I want to use Avro, is it enough to just add
env.getConfig().enableForceAvro();
or do I need to write a custom serializer as well?
a
To disable Kryo and automatic creation of UID:
Copy code
Configuration config = new Configuration();
config.set(PipelineOptions.GENERIC_TYPES, false); // env.getConfig().disableGenericTypes(); could also be used, but it's deprecated in favor of the GENERIC_TYPES setting
config.set(PipelineOptions.AUTO_GENERATE_UIDS, false);

StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment(config)
The GENERIC_TYPES = false setting disables Kryo entirely and will make Flink throw an UnsupportedOperationException whenever it encounters a data type that would go through Kryo for serialization. As @Pedro Mázala mentions disabling auto generated UIDs is also good practice. From the docs:
If you do not specify the IDs manually they will be generated automatically. You can automatically restore from the savepoint as long as these IDs do not change. The generated IDs depend on the structure of your program and are sensitive to program changes. Therefore, it is highly recommended assigning these IDs manually.
The effectively means that the state will become un-restorable after certain program changes (because the auto-assigned UIDs have changed). So always manually assign UIDs
☝️ 1
👍 1
m
So by setting these two configurations, I don't need anything else, and Kryo won't be used anymore, and POJO will always be used? Or do I still need to register my types?
a
You shouldn't need to register POJO types. However, you probably need to change your types to be POJO compatible. Flink will automatically prefer using POJO serialization before resorting to Kryo. So some of your types are not POJO compatible. See details here: https://nightlies.apache.org/flink/flink-docs-master/docs/dev/datastream/fault-tolerance/serialization/types_serialization/#pojos
❤️ 1
m
thanksssss so much
a
No problem. Good luck. I hope it works out
❤️ 1