Hi Everyone, I am new to flink. I am exploring to use flink stateful functions in production for workflow orchestration. Is stateful functions recommended to use in production? Are there any teams using it in production?

We aren’t yet running stateful functions in production, but are using them to implement was is basically stateful microservices. If that’s what you need (message passing between “services”), plus good integration with services written in other languages (like Python), and/or iterations (one operator/service feeding events upstream) then stateful functions are a win. Otherwise for standard ETL-ish stuff I’dd stick with DataStreams.

thanks Ken. my main need is somewhat similar to “mesage passing between microservices”. I have need to implement workflow, where i can call various microservices to perform various steps for each item. I was thinking for stateful functions as it would help to persist state of the workflow, and also track and control flow of each item.

If you’re just doing a lot of calls to external microservices, then I think a regular Flink DataStream workflow with lots of AsyncFunctions to connect to these microservices would work best. See https://nightlies.apache.org/flink/flink-docs-master/docs/dev/datastream/operators/asyncio/ for more details

thanks ken

You can use two Kafka topics as an API, yes. So nothing special about that, other than (in general) properly configuring your Kafka producers & consumers is more complex than using an HTTP or JDBC API. You also have some additional work to ensure you correctly match up your Kafka response to the request, and handling cases where you are restarting the workflow (and thus need to ignore some responses to in-flight requests), etc. Not trivial, but doable.

i think, i got your idea, i can map kafka as input to async function and perform the logic based on the message i receive