Found problem, fix in code, cancel job, write last...
# troubleshooting
n
Found problem, fix in code, cancel job, write last checkpoint path, reupload jar, submit job again with same args and with last checkpoint path as savepoint. But got StateMigrationException:
Copy code
java.lang.RuntimeException: Error while getting state
	at org.apache.flink.runtime.state.DefaultKeyedStateStore.getMapState(DefaultKeyedStateStore.java:109)
	at org.apache.flink.streaming.api.operators.StreamingRuntimeContext.getMapState(StreamingRuntimeContext.java:242)
	at xxx.xxx.xxx.flink.jobs.function.xxxxFunction.open(xxxxFunction.java:63)
	at org.apache.flink.api.common.functions.util.FunctionUtils.openFunction(FunctionUtils.java:34)
	at org.apache.flink.streaming.api.operators.AbstractUdfStreamOperator.open(AbstractUdfStreamOperator.java:102)
	at org.apache.flink.streaming.api.operators.KeyedProcessOperator.open(KeyedProcessOperator.java:55)
	at org.apache.flink.streaming.runtime.tasks.RegularOperatorChain.initializeStateAndOpenOperators(RegularOperatorChain.java:107)
	at org.apache.flink.streaming.runtime.tasks.StreamTask.restoreStateAndGates(StreamTask.java:858)
	at org.apache.flink.streaming.runtime.tasks.StreamTask.lambda$restoreInternal$5(StreamTask.java:812)
	at org.apache.flink.streaming.runtime.tasks.StreamTaskActionExecutor$1.call(StreamTaskActionExecutor.java:55)
	at org.apache.flink.streaming.runtime.tasks.StreamTask.restoreInternal(StreamTask.java:812)
	at org.apache.flink.streaming.runtime.tasks.StreamTask.restore(StreamTask.java:771)
	at org.apache.flink.runtime.taskmanager.Task.runWithSystemExitMonitoring(Task.java:970)
	at org.apache.flink.runtime.taskmanager.Task.restoreAndInvoke(Task.java:939)
	at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:763)
	at org.apache.flink.runtime.taskmanager.Task.run(Task.java:575)
	at java.base/java.lang.Thread.run(Unknown Source)
Caused by: org.apache.flink.util.StateMigrationException: For heap backends, the new state serializer (org.apache.flink.api.common.typeutils.base.MapSerializer@69dc107d) must not be incompatible with the old state serializer (org.apache.flink.api.common.typeutils.base.MapSerializer@b3809c67).
	at org.apache.flink.runtime.state.heap.HeapKeyedStateBackend.tryRegisterStateTable(HeapKeyedStateBackend.java:248)
	at org.apache.flink.runtime.state.heap.HeapKeyedStateBackend.createOrUpdateInternalState(HeapKeyedStateBackend.java:327)
	at org.apache.flink.runtime.state.heap.HeapKeyedStateBackend.createOrUpdateInternalState(HeapKeyedStateBackend.java:314)
	at org.apache.flink.runtime.state.KeyedStateFactory.createOrUpdateInternalState(KeyedStateFactory.java:47)
	at org.apache.flink.runtime.state.ttl.TtlStateFactory.createStateAndWrapWithTtlIfEnabled(TtlStateFactory.java:70)
	at org.apache.flink.runtime.state.AbstractKeyedStateBackend.getOrCreateKeyedState(AbstractKeyedStateBackend.java:361)
	at org.apache.flink.runtime.state.AbstractKeyedStateBackend.getPartitionedState(AbstractKeyedStateBackend.java:412)
	at org.apache.flink.runtime.state.DefaultKeyedStateStore.getPartitionedState(DefaultKeyedStateStore.java:115)
	at org.apache.flink.runtime.state.DefaultKeyedStateStore.getMapState(DefaultKeyedStateStore.java:106)
	... 16 more
In code I've added few debug logs to xxxxFunction.process() so nothing changed to state store. When I run same job but locally with same checkpoint on same code all works well. Whats problem and what to do? State descriptor:
Copy code
MapStateDescriptor<String, List<RElement<MValue>>> subEventsDescriptor = new MapStateDescriptor<>(
		"re_mv_j",
		BasicTypeInfo.STRING_TYPE_INFO,
		new ListTypeInfo<>(TypeInformation.of(new TypeHint<>() {
		}))
);
l
What version of Flink are you using? Did you update the Flink version? Are you explicitly setting the operator UIDs / did you set
pipeline.auto-generate-uids
to false? The error makes it sound like something about the state changed between what the checkpoint file is holding versus what the new job graph is expecting - which could be due to the operator UIDs being regenerated. Also, if you don't do this now, then it might be a good idea to create a savepoint when cancelling just in case the last checkpoint wasn't successful. If you have an issue reloading the state, and if the state isn't critical to maintain, then you can also try restarting with the
allowNonRestoredState
option.