Zeyu Qiu
09/01/2025, 8:02 AMparallelism = 1
env = StreamExecutionEnvironment.get_execution_environment(config)
env.set_parallelism(parallelism) # I don't know what's that, try 2
env.enable_checkpointing(10 * 60 * 1000) # milliseconds
# checkpoints have to complete within one minute, or are discarded
env.get_checkpoint_config().set_checkpoint_timeout(int(10 * 60 * 1000))
env.get_checkpoint_config().set_checkpointing_mode(
CheckpointingMode.EXACTLY_ONCE
)
env.disable_operator_chaining() # If drop this line, create multiple pipeline in one python job will raise error in Flink GUI
settings = EnvironmentSettings.new_instance().in_streaming_mode().build()
t_env = StreamTableEnvironment.create(env, environment_settings=settings)
ss = t_env.create_statement_set()
source_sql = f"""
CREATE TABLE mysql_source (
db_name STRING METADATA FROM 'database_name' VIRTUAL,
table_name STRING METADATA FROM 'table_name' VIRTUAL,
id BIGINT,
content STRING,
update_time TIMESTAMP(3)
) WITH (
'connector' = 'mysql-cdc',
'hostname' = '{db_secret["host"]}',
'port' = '{db_secret["port"]}',
'username' = '{db_secret["username"]}',
'password' = '{db_secret["password"]}',
'database-name' = 'my_db',
'table-name' = 'my_table',
'server-id' = '{server_id}',
'debezium.snapshot.mode' = 'schema_only',
'scan.startup.mode' = 'timestamp',
'scan.startup.timestamp-millis' = '{binlog_start_time}',
'scan.incremental.snapshot.enabled' = 'true',
'scan.incremental.snapshot.chunk.key-column' = 'id'
);
"""
t_env.execute_sql(source_sql)
sink_sql = f"""
CREATE TABLE hudi_sink (
id BIGINT,
content STRING,
update_time TIMESTAMP(3),
hudi_ts double,
PRIMARY KEY (id) NOT ENFORCED
) WITH (
'connector' = 'hudi',
'path' = '<s3a://xxxx>',
'table.type' = 'COPY_ON_WRITE',
'write.precombine.field' = 'hudi_ts',
'write.operation' = 'upsert',
'hoodie.datasource.write.recordkey.field' = 'id',
'hoodie.datasource.write.partitionpath.field' = '',
'write.tasks' = '1',
'compaction.tasks' = '1',
'clean.retain_commits' = '6',
'hoodie.keep.min.commits' = '7',
'hoodie.keep.max.commits' = '8',
'compaction.async.enabled' = 'true',
'hoodie.write.lock.provider' = 'org.apache.hudi.client.transaction.lock.InProcessLockProvider',
'index.partition.regex' = 'false',
'index.bootstrap.enabled' = 'false',
'write.rate.limit' = '2000'
);
"""
t_env.execute_sql(sink_sql)
insert_sql = f"""
INSERT INTO hudi_sink
SELECT id,content,update_time,
UNIX_TIMESTAMP() as hudi_ts FROM mysql_source where db_name = 'my_db'
and table_name = 'my_table';
"""
ss.add_insert_sql(insert_sql)
ss.attach_as_datastream()
env.execute(f'mysql_cdc_to_hudi')
After I submit this job to Flink, I expect the Hudi table in S3 could have real-time data same as MySQL table. But in fact, when I execute an update statement in MySQL, there will be 2 duplicate rows appear in Hudi table. For example here’s data in MySQL:
id |content |update_time
1 |a |2025-08-01 00:00:00
And I execute:
update my_table set content = 'b' where id = 1
Then the data in Hudi table will looks like that:
id |content |update_time
1 |a |2025-08-01 00:00:00
1 |b |2025-09-01 00:00:00
Do you guys have any idea about why that’s happening? As I’ve set 'hoodie.datasource.write.recordkey.field' = 'id' , I suppose there shouldn’t be duplicate row in the Hudi table. And this error cannot be reproduced stably. Sometimes the result is normal, and sometimes there is an error. This is especially likely to occur when there are multiple Hudi sinks in a job.
I’m using Flink 1.16.2 , Hudi 0.13.0 , FlinkCDC 3.1.1
In addition, I found some log in Flink like:
2025-09-01 01:10:20,612 INFO org.apache.flink.table.runtime.operators.sink.SinkUpsertMaterializer [] - The state is cleared because of state ttl. This will result in incorrect result. You can increase the state ttl to avoid this.
Not sure if this related to the problem I met