Hey Flinkers, Is there a recommendation for handling namespace changes for objects in state? For instance, I have a class that is stored in state and defined in state descriptors as

com.someproject.widget.WidgetClass

and I now have a need to share this class in another Flink job so I attempted to migrate it to a separate “common” project. Now the namespacing changed, the class uses

com.common.widget.WidgetClass

. When attempting to upgrade the job, I noticed continual failures as the job attempted to restore state using the previous, now non-existant, class:

Caused by: org.apache.flink.runtime.state.BackendBuildingException: Failed when trying to restore operator state backend
	at org.apache.flink.runtime.state.DefaultOperatorStateBackendBuilder.build(DefaultOperatorStateBackendBuilder.java:83)
	at org.apache.flink.contrib.streaming.state.EmbeddedRocksDBStateBackend.createOperatorStateBackend(EmbeddedRocksDBStateBackend.java:503)
	at org.apache.flink.streaming.api.operators.StreamTaskStateInitializerImpl.lambda$operatorStateBackend$0(StreamTaskStateInitializerImpl.java:277)
	at org.apache.flink.streaming.api.operators.BackendRestorerProcedure.attemptCreateAndRestore(BackendRestorerProcedure.java:168)
	at org.apache.flink.streaming.api.operators.BackendRestorerProcedure.createAndRestore(BackendRestorerProcedure.java:135)
	... 13 more
Caused by: java.io.IOException: Could not find class 'com.someproject.widget.WidgetClass' in classpath.
	at org.apache.flink.util.InstantiationUtil.resolveClassByName(InstantiationUtil.java:775)
	at org.apache.flink.util.InstantiationUtil.resolveClassByName(InstantiationUtil.java:750)

Any recommendations would be welcome as I have to imagine this can occasionally happen. I’m guessing since this is technically a breaking schema change per the Flink docs (i.e. namespace cannot change), I’d likely need to perform some process to either rebootstrap the state, but maybe there’s a alternate way to handle it.

A naive way to handle this would be to move your

WidgetClass

to the new

common

package. Then, create a subclass of it in the existing project.

package com.someproject.widget.WidgetClass

class WidgetClass extends com.common.widget.WidgetClass {}

cc: @Moustapha Mahfoud

if the state inside the WidgetClass isn't important, you can change the UID of the operator

Yeah, in this case I have messages in flight that need the state so unfortunately I can lose it. Also the classes themselves are Kotlin data classes (which are final by default and cannot be inherited).