Hi all! I currently have a Flink job with a series of functions that rely on Broadcast State for enriching data that flow through the job from a variety of streams. In the interest of resiliency and ensuring that all of the records are enriched if possible, I was wondering is it possible to throw an exception in the job if the Broadcast state is empty, preferably after a state restore has been done? I’m trying to mitigate situations that might require some type of manual intervention should important state like Broadcast state become unavailable. I didn’t see any overrides for something like restore/initializeState, so I thought I’d reach out on how I might accomplish something like this (if possible). I think the hardest part would be differentiating between a fresh start of the job and it after a restoration (which I assumed is where the restore/initialize functionality would come in). Any suggestions/advice would be welcome!

Hi! You can have your BroadcastProcessFunction implement CheckpointedFunction, and then you’ll be able to implement initializeState, and use isRestored to figure out what’s going on.

That’s exactly the type of idea I was looking for. Thanks for the advice David, I appreciate it!

You might also be able to bootstrap the broadcast state in the initializeState method by loading it from somewhere.