Hi Flink Community :wave:, I am seeking assistance...
# troubleshooting
b
Hi Flink Community 👋, I am seeking assistance with a data discrepancy issue that I am facing. I am using the Flink Kubernetes Operator to submit a Flink SQL job via PyFlink that consumes from a source Kafka topic and writes to a sink Kafka topic. The source topic is read using the
kafka
connector (https://nightlies.apache.org/flink/flink-docs-release-1.18/docs/connectors/table/kafka/), and the sink topic is written to using the
upsert-kafka
connector (https://nightlies.apache.org/flink/flink-docs-release-1.18/docs/connectors/table/upsert-kafka/). The source topic contains CDC records that are produced to by a Debezium connector. When analyzing the data within the sink topics, I found that there are missing records and I would like to understand why this is the case. Here are some additional details about the Flink application: • State Backend:
RocksDB
• External Checkpoint source:
s3
• Checkpointing Interval:
1h
• Savepoint Interval:
12h
Here is an example of the Flink SQL that is being submitted:
Copy code
CREATE TABLE source (
    `kafka_key_id` INT, 
    `field_1` type,
    `field_2` type,
    ...
    `field_n` type,
    PRIMARY KEY (`kafka_key_id`) NOT ENFORCED
) WITH (
    'key.fields-prefix' = 'kafka_key_',
    'topic' = 'source_topic',
    'scan.startup.mode' = 'earliest-offset',
    'connector' = 'kafka',
    'key.fields' = 'kafka_key_id',
    'key.format' = 'avro-confluent',
    'value.format' = 'debezium-avro-confluent'
)

CREATE TABLE sink (
    `kafka_key_id` INT, 
    `field_1` type,
    `field_2` type,
    ...
    `field_n` type,
    PRIMARY KEY (kafka_key_id) NOT ENFORCED
) WITH (
    'key.fields-prefix' = 'kafka_key_',
    'connector' = 'upsert-kafka',
    'key.format' = 'avro-confluent',
    'value.format' = 'avro-confluent'
)

INSERT INTO sink (
    SELECT
         `kafka_key_id` INT, 
         `field_1` type,
         `field_2` type
          ...
         `field_n` type
    FROM
        source
)
Version Details: • Apache Flink:
1.18
• Flink Kubernetes Operator Version:
1.9.0
• Apache Kafka:
3.5.1
running on Amazon MSK • Debezium:
2.4.0
I found this presentation online:

https://youtu.be/3R8aAj9peTc?si=4LZe9IE3M3tkU2hE▾

(only the first 9 minutes seem to apply) by @rmetzger & @Sharon Xie where the potential of data loss is discussed. Based on the talk, I have the following questions: 1. Under what circumstances does Flink use exactly-once semantics (i.e. the Producer uses transactions to produce messages to ensure that messages are produced only once)? 2. When using Flink to write records into Kafka with the
upsert-kafka
connector and checkpointing enabled, does Flink use exactly-once semantics to produce to Kafka? 3. If the above is true, and Flink is producing to Kafka using exactly-once semantics, then there is a potential for data loss, as mentioned in the presentation. However, is this only true if the Kafka consumer is reading with the
isolation.level
set to
read_committed
? If so, then I don't understand why I am missing records in the sink Kafka topic because I used the
kafka-console-consumer
CLI tool to analyze the missing records, and I was not explicitly setting the
isolation.level
. I also sinked the data from Kafka into Snowflake via the Snowflake Sink Connector (which uses Kafka Connect) to analyze the missing data and did not override the
isolation.level
as mentioned here: https://kafka.apache.org/documentation/#connect_exactlyoncesink. Can I get clarification here? 4. Can I get clarification on the relationship between Flink checkpointing and Kafka Producer Transactions? 5. What is the recommended checkpoint interval when optimizing for minimal latency? Thanks in advance! @Ryan D'Souza @Sandip Nayak @Duncan Prince @Zach Walker @Pankaj Joshi
👍 1