Abhishek Joshi
04/23/2025, 1:31 PM//Kafka Source:
.setStartingOffsets(OffsetsInitializer.committedOffsets(OffsetResetStrategy.EARLIEST));
source.kafka.enable.auto.commit = false;
//Kafka Sink:
.setDeliveryGuarantee(DeliveryGuarantee.EXACTLY_ONCE);
.setTransactionalIdPrefix("event-flow-trigger-mapping");
sink.kafka.isolation.level = read_committed;
sink.kafka.enable.idempotence = true;
sink.kafka.transaction.timeout.ms = 900000;
//Checkpointing:
app.checkpoint.interval.ms = 120000;
app.checkpoint.min-pause = 60000;
env.getCheckpointConfig().setCheckpointingMode(CheckpointingMode.EXACTLY_ONCE);
env.getCheckpointConfig().setCheckpointStorage(path);
env.getCheckpointConfig().setMaxConcurrentCheckpoints(1);
env.getCheckpointConfig().setExternalizedCheckpointCleanup(
ExternalizedCheckpointCleanup.RETAIN_ON_CANCELLATION
);
Has anyone experienced this kind of data loss even with exactly-once semantics enabled? Are there any known edge cases where this might happen during abrupt TaskManager shutdowns? Any pointers or insights would be really helpful.
Thanks!
cc: @Dheeraj PanangatPedro Mázala
04/23/2025, 2:59 PMAbhishek Joshi
04/23/2025, 3:29 PMkafka-console-consumer.sh and applying a filter on the stream based on the traceId field.
Note: here consumer which I used have isolation.level=read_committedPedro Mázala
04/24/2025, 9:16 AMIf the time between Flink application crash and completed restart is > larger than Kafka’s transaction timeout there will be data loss (Kafka will automatically abort transactions that exceeded timeout time).What is your kafka transaction timeout?
Pedro Mázala
04/24/2025, 9:18 AMAbhishek Joshi
04/24/2025, 10:36 AMArvid Heise
04/24/2025, 11:02 AMAbhishek Joshi
04/24/2025, 12:12 PMbin/kafka-console-consumer.sh --bootstrap-server "$BS" --consumer.config "$KAFKA_CLIENT_PROPERTIES" --topic event_flow_mapping-temp --consumer-property isolation.level=read_committed | grep -o '"nsTraceId":"i-rama-5ga4-aj999-anuj[^"]*"' > /home/user/exactlyonce.txt
To stop application I am using kube commands like _kubectl delete pod pod_name_ and deleted task manager's pods. My job is continuously running and I am just deleting TMs in between the event processing to test this exactly once scenario as I wanted to achieve even though there is any failure or restart or reprocessing happened duplicate events should not go to downstream pipeline and without expecting any data loss.Arvid Heise
04/24/2025, 12:24 PMAbhishek Joshi
04/24/2025, 12:29 PMArvid Heise
04/24/2025, 12:39 PMPedro Mázala
04/24/2025, 12:53 PMAbhishek Joshi
04/24/2025, 8:05 PMsource kafka --> Flink App with Exactly once setup --> Sink kafka --> read_committed consumer --> Another app which is consuming committed only events
And yeah I checked application there was no failure or event loss during processing from app side, so then I should be produced and if they consumed--> produced and committed (even reconsumed and reprocessed the events from last checkpoint due to TM failure) then why there is the difference in source and sink events with exactly once.Abhishek Joshi
04/24/2025, 8:06 PMArvid Heise
04/25/2025, 6:57 AMArvid Heise
04/25/2025, 6:57 AMArvid Heise
04/25/2025, 6:58 AMAbhishek Joshi
04/25/2025, 8:47 AMArvid Heise
04/25/2025, 10:59 AMAbhishek Joshi
04/26/2025, 6:55 PMcommitTransaction(), right?
If my understanding is correct, what would happen if TaskManagers (TMs) are deleted after checkpoint completion but before the transaction commit phase?
Would the events/data that are already in the Kafka transaction buffer be lost?
Could this be one of the reasons for minor data loss even when exactly-once is configured properly?
If so, how can we handle or avoid this critical edge case?
Because if transaction commits are not properly fault-tolerant after checkpoint completion, then technically there can still be some gaps, which is very important from a strong exactly-once guarantee perspective.Arvid Heise
04/26/2025, 7:42 PMAbhishek Joshi
04/26/2025, 9:40 PMDanny Cranmer
04/27/2025, 8:21 AMDanny Cranmer
04/27/2025, 8:23 AMAbhishek Joshi
04/27/2025, 8:27 AMAbhishek Joshi
04/28/2025, 9:02 AMArvid Heise
04/28/2025, 10:35 AMArvid Heise
04/28/2025, 10:37 AMArvid Heise
04/28/2025, 10:38 AM