what's the flink way of passing over the same data multiple times in a stream? Putting the stream in a UDF and for loops seems like the wrong solution.

side output to a kafka topic and consume that topic

😢 I don't use kafka just flink

then the udf seems ok, encapsulate the busines logic in some function

I see, so then assuming I have a looping statement in there. How much is too much work to do in my UDF? Are there any guidelines?

You mean like an iteration? https://nightlies.apache.org/flink/flink-ml-docs-release-2.0/docs/development/iteration/

Hmm interesting, although i'm not using Flink ML, and i'm operating entirely in pyflink

Iterations in a regular streaming dataflow program is a not a thing to do: the entire jobgraph is meant to be acyclic

and yet... the data science guys want to use iterations to perform multiple passes over a data set

For ML it makes sense: that’s why it’s in Flink ML 😅

thanks guys, going to do a bit more reading and come up with a strategy