Evan NotEvan
10/06/2025, 11:36 AM1024, the output sequence should be:
1024 -> 512 -> 256 -> ... -> 1
The interesting part is that I’m using Flink 2.1, where DataStream.iterate() and `IterativeStream`—basically the built-in Iteration API—have been removed. On top of that, Flink ML currently only supports Flink versions up to 1.17.
Without these options, it’s up to the user to implement their own iteration. So far, I’ve put together the following code, which attempts to demonstrate a feedback loop. I’ve also included a diagram to better illustrate the implementation.
import org.apache.flink.api.common.eventtime.WatermarkStrategy;
import org.apache.flink.api.common.functions.FilterFunction;
import org.apache.flink.api.common.functions.MapFunction;
import org.apache.flink.streaming.api.datastream.DataStream;
import org.apache.flink.streaming.api.datastream.SingleOutputStreamOperator;
import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
import org.apache.flink.streaming.api.functions.ProcessFunction;
import org.apache.flink.util.Collector;
import org.apache.flink.util.OutputTag;
public class FeedbackLoopExample {
// Define a side output for feedback
private static final OutputTag<Integer> FEEDBACK_TAG = new OutputTag<Integer>("feedback"){};
public static void main(String[] args) throws Exception {
final StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
// Initial input
DataStream<Integer> input = env.fromElements(1024)
.assignTimestampsAndWatermarks(WatermarkStrategy.noWatermarks());
// Process function that splits into converged vs non-converged
SingleOutputStreamOperator<Integer> processed = input.process(new ProcessFunction<Integer, Integer>() {
@Override
public void processElement(Integer value, Context ctx, Collector<Integer> out) {
if (value > 1) {
// Not yet converged → send to feedback
ctx.output(FEEDBACK_TAG, value / 2); // iteration step
} else {
// Converged → emit as output
out.collect(value);
}
}
});
// Feedback stream (non-converged)
DataStream<Integer> feedback = processed.getSideOutput(FEEDBACK_TAG);
// Apply the same process again (iterative feedback)
SingleOutputStreamOperator<Integer> processedFeedback = feedback.process(new ProcessFunction<Integer, Integer>() {
@Override
public void processElement(Integer value, Context ctx, Collector<Integer> out) {
if (value > 1) {
ctx.output(FEEDBACK_TAG, value / 2);
} else {
out.collect(value);
}
}
});
// Union feedback outputs → continue iteration
DataStream<Integer> loopFeedback = processedFeedback.getSideOutput(FEEDBACK_TAG);
// Final results = outputs from both initial and feedback iterations
DataStream<Integer> finalResults = processed.union(processedFeedback);
finalResults.print("Converged");
loopFeedback.print("Feedback"); // shows values still circulating
env.execute("Flink Feedback Loop Example");
}
}
This is the output if you run the code
1024 -> 512 -> 256 (stops here)
It seems like the job only performs two iterations and then stops instead of counting down to 1. This makes me think that my attempt to loop back the feedback stream, didn’t actually work.
My first question is: what’s the correct way to implement a feedback loop in Flink 2.0+ given the constrains mentioned above?
Second question: is it ok if two different process functions use the same side out to write data ?
Third question; is there any way to extract or analyze the graph topology of a job in Flink ?