Hi all. We have a streaming job that runs continuously. We need to be able to dynamically extract old data from a data warehouse and then process it in batch mode and then merge the resulting state into the already running streaming job. Is there a way to achieve that?

the state processing API might be able to help

You could have the batch job generate files to a directory that a FileSource in your streaming job is monitoring, and that becomes a stream that gets merged where appropriate to update state in your streaming job.

thanks Ken. I'd need to find a way to encode the state in the files, right?

Correct - but “encode the state” could just mean a simple text file format with key<tab>value (as a very simple example)

makes sense. Thanks!