Hello Everyone, New to Flink and also to realtime data processing system. In our Flink application(Kafka -> Flink App -> Postgres & BigQuery -> Kafka) we want to ensure atleast once processing of the kafka events. Checkpointing is enabled, so whenever there is an un-handled exception in the code, the Flink retries and process the messages from the last checkpoint and meanwhile i need to deploy the bug fix and flink start processing from the last checkpoint, so there is no event loss. Is my understanding correct?

To avoid loosing the events, i need to throw exception in all the places where the operation didn’t behave as expected ex: not able to write the data into table or some dependent data is missing and i need to wait etc

Yes, with checkpointing enabled you should expect at-least-once delivery (at least 🙂 some can do exactly-once by default) in most of the Flink connectors. However, there are some edge-case scenarios, e.g. if you use Kafka sink with exactly-once enabled you need to tune the Kafka cluster (https://docs.immerok.cloud/docs/cookbook/exactly-once-with-apache-kafka-and-apache-flink/#prevent-transaction-timeouts). Just read the docs for each connector that you use.

Frantically trying to fix a bug before a streaming job re-processes the record that triggered the exception is not much fun 🙂 Typically I send bad records to side output in a ProcessFunction that validates incoming data, and that side output gets written to a Kafka topic. When the bug is fixed, I take a savepoint, stop the workflow, redeploy the new version, and this time run it in a mode where it also consumes from the “bad data” topic. Once all records in that have been processed, I savepoint/stop again, and restart w/o the bad topic. I could likely wrap or configure the bad data Kafka source to make it bounded, which would save the second stop/start, but that’s work for another day…

Hi @Ken Krugler,Thank you very much for taking time in replying. That’s super interesting and would definitely explore and try. Is this kind of automated like part of your CI/CD pipeline where you pass some parameters and the logic to include bad data topic etc? Thanks again

It’s not automated, but it’s baked into the tool (main entry point) of the app. E.g. something like if the optional

-baddata <topic>

parameter is passed on the command line, then hook that up as a source to the workflow.

Got it, thanks again