Hi, I have a use case when I need to preload some data from multiple Kafka Streams for a broadcast state and use it as enrichment for the main stream. Is it possible to delay main stream startup until the data preloads?

As far as I know, that's not possible

Aw pity, thanks!

Actually, there is a way to do this. You'll need to implement the CheckpointedFunction interface. Then the initializeState method you can check to see if the job is being restored, in which you probably already have the necessary broadcast data coming from a checkpoint, and if not, you can load that data from wherever.

But would that hold off loading the startup during initial load? My understanding is that it’s desired to first get something like rules loaded, before the actual processing of the data with those rules active needs to happen

initializeState is part of the job warmup phase that gets completed before events begin to flow.

TIL 🙂 Thanks David

Note that you can not use this trick to preload keyed state. This only works with operator state, like broadcast state.

I see that. The problem I'm trying to solve is preloading data that rarely changes to memory. Or at least keeping it in memory as long as possible. Perhaps there are some other strategies to do so? I'm a bit cautious about state because I the cluster I work with is deployed to kubernetes without any storage. 😕