Hello, I am having trouble implementing an iterati...
# troubleshooting
e
Hello, I am having trouble implementing an iterative computation in Flink via feedback loop. I’ve prepared a sample code and a diagram to illustrate the idea. The concept is straightforward: - Take an input number. - Repeatedly divide it by 2 until the result is less than or equal to 1. - Along the way, print all intermediate results. For example, given the input
1024
, the output sequence should be:
Copy code
1024 -> 512 -> 256 -> ... -> 1
The interesting part is that I’m using Flink 2.1, where
DataStream.iterate()
and `IterativeStream`—basically the built-in Iteration API—have been removed. On top of that, Flink ML currently only supports Flink versions up to 1.17. Without these options, it’s up to the user to implement their own iteration. So far, I’ve put together the following code, which attempts to demonstrate a feedback loop. I’ve also included a diagram to better illustrate the implementation.
Copy code
import org.apache.flink.api.common.eventtime.WatermarkStrategy;
import org.apache.flink.api.common.functions.FilterFunction;
import org.apache.flink.api.common.functions.MapFunction;
import org.apache.flink.streaming.api.datastream.DataStream;
import org.apache.flink.streaming.api.datastream.SingleOutputStreamOperator;
import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
import org.apache.flink.streaming.api.functions.ProcessFunction;
import org.apache.flink.util.Collector;
import org.apache.flink.util.OutputTag;

public class FeedbackLoopExample {

    // Define a side output for feedback
    private static final OutputTag<Integer> FEEDBACK_TAG = new OutputTag<Integer>("feedback"){};

    public static void main(String[] args) throws Exception {
        final StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();

        // Initial input
        DataStream<Integer> input = env.fromElements(1024)
                .assignTimestampsAndWatermarks(WatermarkStrategy.noWatermarks());

        // Process function that splits into converged vs non-converged
        SingleOutputStreamOperator<Integer> processed = input.process(new ProcessFunction<Integer, Integer>() {
            @Override
            public void processElement(Integer value, Context ctx, Collector<Integer> out) {
                if (value > 1) {
                    // Not yet converged → send to feedback
                    ctx.output(FEEDBACK_TAG, value / 2);  // iteration step
                } else {
                    // Converged → emit as output
                    out.collect(value);
                }
            }
        });

        // Feedback stream (non-converged)
        DataStream<Integer> feedback = processed.getSideOutput(FEEDBACK_TAG);

        // Apply the same process again (iterative feedback)
        SingleOutputStreamOperator<Integer> processedFeedback = feedback.process(new ProcessFunction<Integer, Integer>() {
            @Override
            public void processElement(Integer value, Context ctx, Collector<Integer> out) {
                if (value > 1) {
                    ctx.output(FEEDBACK_TAG, value / 2);
                } else {
                    out.collect(value);
                }
            }
        });

        // Union feedback outputs → continue iteration
        DataStream<Integer> loopFeedback = processedFeedback.getSideOutput(FEEDBACK_TAG);

        // Final results = outputs from both initial and feedback iterations
        DataStream<Integer> finalResults = processed.union(processedFeedback);

        finalResults.print("Converged");
        loopFeedback.print("Feedback"); // shows values still circulating

        env.execute("Flink Feedback Loop Example");
    }
}
This is the output if you run the code
Copy code
1024 -> 512 -> 256 (stops here)
It seems like the job only performs two iterations and then stops instead of counting down to 1. This makes me think that my attempt to loop back the feedback stream, didn’t actually work. My first question is: what’s the correct way to implement a feedback loop in Flink 2.0+ given the constrains mentioned above? Second question: is it ok if two different process functions use the same side out to write data ? Third question; is there any way to extract or analyze the graph topology of a job in Flink ?