Brandon Stanley
10/12/2024, 5:11 PMkafka connector (https://nightlies.apache.org/flink/flink-docs-release-1.18/docs/connectors/table/kafka/), and the sink topic is written to using the upsert-kafka connector (https://nightlies.apache.org/flink/flink-docs-release-1.18/docs/connectors/table/upsert-kafka/). The source topic contains CDC records that are produced to by a Debezium connector. When analyzing the data within the sink topics, I found that there are missing records and I would like to understand why this is the case.
Here are some additional details about the Flink application:
• State Backend: RocksDB
• External Checkpoint source: s3
• Checkpointing Interval: 1h
• Savepoint Interval: 12h
Here is an example of the Flink SQL that is being submitted:
CREATE TABLE source (
`kafka_key_id` INT,
`field_1` type,
`field_2` type,
...
`field_n` type,
PRIMARY KEY (`kafka_key_id`) NOT ENFORCED
) WITH (
'key.fields-prefix' = 'kafka_key_',
'topic' = 'source_topic',
'scan.startup.mode' = 'earliest-offset',
'connector' = 'kafka',
'key.fields' = 'kafka_key_id',
'key.format' = 'avro-confluent',
'value.format' = 'debezium-avro-confluent'
)
CREATE TABLE sink (
`kafka_key_id` INT,
`field_1` type,
`field_2` type,
...
`field_n` type,
PRIMARY KEY (kafka_key_id) NOT ENFORCED
) WITH (
'key.fields-prefix' = 'kafka_key_',
'connector' = 'upsert-kafka',
'key.format' = 'avro-confluent',
'value.format' = 'avro-confluent'
)
INSERT INTO sink (
SELECT
`kafka_key_id` INT,
`field_1` type,
`field_2` type
...
`field_n` type
FROM
source
)
Version Details:
• Apache Flink: 1.18
• Flink Kubernetes Operator Version: 1.9.0
• Apache Kafka: 3.5.1 running on Amazon MSK
• Debezium: 2.4.0
I found this presentation online: (only the first 9 minutes seem to apply) by @rmetzger & @Sharon Xie where the potential of data loss is discussed. Based on the talk, I have the following questions:
1. Under what circumstances does Flink use exactly-once semantics (i.e. the Producer uses transactions to produce messages to ensure that messages are produced only once)?
2. When using Flink to write records into Kafka with the upsert-kafka connector and checkpointing enabled, does Flink use exactly-once semantics to produce to Kafka?
3. If the above is true, and Flink is producing to Kafka using exactly-once semantics, then there is a potential for data loss, as mentioned in the presentation. However, is this only true if the Kafka consumer is reading with the isolation.level set to read_committed? If so, then I don't understand why I am missing records in the sink Kafka topic because I used the kafka-console-consumer CLI tool to analyze the missing records, and I was not explicitly setting the isolation.level. I also sinked the data from Kafka into Snowflake via the Snowflake Sink Connector (which uses Kafka Connect) to analyze the missing data and did not override the isolation.level as mentioned here: https://kafka.apache.org/documentation/#connect_exactlyoncesink. Can I get clarification here?
4. Can I get clarification on the relationship between Flink checkpointing and Kafka Producer Transactions?
5. What is the recommended checkpoint interval when optimizing for minimal latency?
Thanks in advance!
@Ryan D'Souza @Sandip Nayak @Duncan Prince @Zach Walker @Pankaj Joshi