Hi all, I’m testing exactly-once semantics in a F...
# troubleshooting
a
Hi all, I’m testing exactly-once semantics in a Flink job with Kafka as both the source and sink. While most of the setup works as expected, I’m noticing a consistent data loss when I forcefully kill all TaskManagers during processing or any failure happened. Test Scenario: Input: 3 million records pushed to Kafka source topic. During processing: All TaskManagers were deleted to simulate failure. Validation: Used a Kafka consumer with isolation.level=read_committed to read from the sink topic. Observed: Input records: 2,999,981 Output records: 2,999,675 Missing: 306 records Given that I'm using exactly-once with checkpoints and transactional sink, I expected Flink to recover and replay all uncommitted data. But it seems like some data is getting lost consistently in this failure scenario. Configurations Used:
Copy code
//Kafka Source:
.setStartingOffsets(OffsetsInitializer.committedOffsets(OffsetResetStrategy.EARLIEST));
source.kafka.enable.auto.commit = false;

//Kafka Sink:
.setDeliveryGuarantee(DeliveryGuarantee.EXACTLY_ONCE);
.setTransactionalIdPrefix("event-flow-trigger-mapping");
sink.kafka.isolation.level = read_committed;
sink.kafka.enable.idempotence = true;
sink.kafka.transaction.timeout.ms = 900000;

//Checkpointing:
app.checkpoint.interval.ms = 120000;
app.checkpoint.min-pause = 60000;
env.getCheckpointConfig().setCheckpointingMode(CheckpointingMode.EXACTLY_ONCE);
env.getCheckpointConfig().setCheckpointStorage(path);
env.getCheckpointConfig().setMaxConcurrentCheckpoints(1);
env.getCheckpointConfig().setExternalizedCheckpointCleanup(
    ExternalizedCheckpointCleanup.RETAIN_ON_CANCELLATION
);
Has anyone experienced this kind of data loss even with exactly-once semantics enabled? Are there any known edge cases where this might happen during abrupt TaskManager shutdowns? Any pointers or insights would be really helpful. Thanks! cc: @Dheeraj Panangat
p
Oof. This is something serious. How are you checking those numbers? When testing again do you have similar output? How are you doing the count of the messages?
a
yes I tested this multiple times and in each iteration I am getting less count at sink topic consumer. I have unique traceId in the event payload and we are counting the messages by consuming them from the Kafka topic (both input and output sink topic ) using the
kafka-console-consumer.sh
and applying a filter on the stream based on the
traceId
field. Note: here consumer which I used have isolation.level=read_committed
p
Fair. This is strange. I just found this:
If the time between Flink application crash and completed restart is > larger than Kafka’s transaction timeout there will be data loss (Kafka will automatically abort transactions that exceeded timeout time).
What is your kafka transaction timeout?
a
I have set kafka.transaction.timeout.ms = 900000 means 15 min and I can see once I delete the TMs it will respawn again within few seconds < transcation timeout time
a
I couple of questions: • Which versions did you use (flink, kafka, connector)? • Why are 19 records missing on the input? How did you get this number? Metrics? • How do you stop the application? • Can you check if them missing records are at the start of the partitions, in the middle, or at the end? • How long is your job running overall? 15min transaction timeout is really low for Flink.
a
Hi Arvid, I am using below version Flink: 1.19 Kafka: 3.7.0 Connector: 3.2.0 You can ignore those 19 events at source, I am using jmeter to push load and I have time constraints so those got missed. As I mentioned I am getting these metrics from kafka by applying grep on traceId and and kafka-consumers commands. Eg:
Copy code
bin/kafka-console-consumer.sh --bootstrap-server "$BS" --consumer.config "$KAFKA_CLIENT_PROPERTIES" --topic event_flow_mapping-temp --consumer-property isolation.level=read_committed | grep -o '"nsTraceId":"i-rama-5ga4-aj999-anuj[^"]*"' > /home/user/exactlyonce.txt
To stop application I am using kube commands like _kubectl delete pod pod_name_ and deleted task manager's pods. My job is continuously running and I am just deleting TMs in between the event processing to test this exactly once scenario as I wanted to achieve even though there is any failure or restart or reprocessing happened duplicate events should not go to downstream pipeline and without expecting any data loss.
a
How do you verify that there are no records still stuck in the pipeline before killing flink?
a
I am not killing the complete application, to simulate failure scenario I am just deleting TMs which will spawn up automatically and resume the processing of remaining events. At the end I am check the lag in pipeline and if there is no lag on downstream consumer/pipeline which means all events processed. Even after all processing my job is still running with newly spawn TMs.
a
no lag means that all events have been consumed but it's no necessarily true that they are all produced
p
Have you checked Kafka's logs to check aborted transactions?
a
Hey Arvid, yeah I agree that no lags means its not necessarily that they were produced but I checked the lag on the another consumer (read_committed) which is consuming from the sink of my application (where we setup the exactly once configs). Pipeline is like
Copy code
source kafka --> Flink App with Exactly once setup --> Sink kafka --> read_committed consumer  --> Another app which is consuming committed only events
And yeah I checked application there was no failure or event loss during processing from app side, so then I should be produced and if they consumed--> produced and committed (even reconsumed and reprocessed the events from last checkpoint due to TM failure) then why there is the difference in source and sink events with exactly once.
Hey Pedro, In kafka logs how can we exactly check for aborted transactions, can you please help here more to understand this?
a
No lag on downstream doesn't help you either to determine if you have processed all data.
You either need to shutdown flink with savepoint with drain or you really need to wait for a longer time to make sure that the last checkpoint finishes and the transaction has been committed.
If you look at what records are missing, my best bet currently is the tail records.
a
Yes I already waited for longer time and then shared my observation here.
a
Can you please check where the missing records appear?
a
Hey Arvid/Pedro, I had one clarification regarding the exactly-once semantics in Flink Kafka producer: Suppose during the pre-commit phase, the producer has already flushed the records into the Kafka transaction buffer (but the transaction is not yet committed). It waits until the checkpoint completion notification to finally call
commitTransaction()
, right? If my understanding is correct, what would happen if TaskManagers (TMs) are deleted after checkpoint completion but before the transaction commit phase? Would the events/data that are already in the Kafka transaction buffer be lost? Could this be one of the reasons for minor data loss even when exactly-once is configured properly? If so, how can we handle or avoid this critical edge case? Because if transaction commits are not properly fault-tolerant after checkpoint completion, then technically there can still be some gaps, which is very important from a strong exactly-once guarantee perspective.
a
Transactions are retried on recovery until they succeed. They may time out in the meantime.
a
But right now when TM shuts down there is one which takes responsibility to commit those transactions right and these transactions not committed then will mot be read by downstream read_committed consumer, right? And when new TMs spawn up they started with checkpoints which is already ahead of as we completed checkpointing phase before TM went down. So new offsets get started and at sink side older events got aborted right? Correct me I am missing something here.
d
Transactions to be committed are written into the checkpoint. When the new TM restores from this checkpoint it will retry committing the transaction.
Are you missing a whole checkpoint worth of data?or a few records?
a
Only few records
Hi Arvid, Danny, and Flink Community, I ran another round of testing, but unfortunately, I still observed data loss. As per the earlier discussions, I understand that if a failure happens after checkpoint completion, the Kafka transactions should be retried for commit once TaskManagers (TMs) restore from the last checkpoint. If that were happening properly, then ideally there shouldn't be any data loss. To rule out transaction timeouts, I already increased the Kafka transaction timeout to 30 minutes, and in my tests, the TMs restored within 1–2 minutes after failure — so timeout shouldn't be the reason here. Given this setup, could you help me understand what might still be causing the data loss? Are there any other failure modes or internal retry behaviors that could explain this? Any insights or directions would be very helpful. Thank you!
a
Since you keep ignoring my question, I cannot help you directly. But I can give you guidance on how I'd approach things: • Figure out which records are missing. You should be able to find that when diffing output and input. Are they spread across partitions? • Figure out when they are sent and at which times they are processed by Flink. Is this near the failure or near the end of the application? • Find the corresponding kafka transactions (prefix-subtaskid-chkid). • Find all logs corresponding to that kafka transaction. • It certainly helps to use DEBUG logging.
A metacomment: Flink and Kafka is rather battletested and is used successfully on much larger scale than you have tested. At least 2 commercial offerings make heavy use of both systems. While there is a chance that there are still bugs, imho it's more likely that your test setup is incorrect. You could also uncover an edge case in your specific setup.
Another comment: Flink is quirky when the stream of data suddenly stops if you have stateful operations. It would be interesting to know your full topology.