Hi everyone, is there any example or guide for chaining two Streaming jobs. We are planning to generate some files in first streaming job output and plug another down streaming file streaming job.

I woudl say that what you need is two separate Flink jobs. The first with FileSink (or any other Sink that can write files) and the second one with FileSource that would read from directory created by the first job.

As @Krzysztof Chmielewski noted, you typically use a sink/source to bridge two Flink jobs. You can also use Kafka as that bridge, as otherwise you would need to deal with cleaning up older files in a FileSink/FileSource approach. I’ve never tried Redpanda as this kind of bridge, but seems promising given its lower ops/CPU overhead when compared to Kafka.

Yeah the challenge we have is the lack of Kafka in middle ans Files management is really a tough job. Read multiple articles and we are planning to try Flink + iceberg to have this file mgmt in middle with incremental processing.