Question: lets say my Flink job faced a failure an...
# randomg
gadhiyakrutik
06/13/2023, 1:42 PMQuestion: lets say my Flink job faced a failure and it tried to restart the job as mentioned in the restart strategy, then
1. will the job start from beginning?
2. will it try to recover with latest checkpoint
3. will it preserver any state data, or broadcast state data
a
Alex Cruise
06/13/2023, 4:53 PMThe job will pick up the last successful checkpoint as its state after restarting, assuming checkpointing was enabled before the crash, and the contents can be deserialized successfully. Many sources will keep state that lets them resume from roughly where they left off, e.g. partition+offset pairs for a Kafka source.
π 1
Alex Cruise
06/13/2023, 4:56 PMI say "roughly" because by default everything is at-least-once, meaning data that was received after the last successful checkpoint would be reprocessed, unless the service you're sourcing from has its own fine-grained session state
Alex Cruise
06/13/2023, 4:57 PMExactly-once is supported in Flink but isn't always possible with all sources + sinks, and can affect throughput and/or latency
2 Views