Hey all, I’m running into an issue. We’re using

flink-cdc-pipeline-connector-iceberg (3.6.0-2.2)

to stream MySQL → Iceberg on Glue. When the partition key column is updated, we’re getting duplicate rows instead of the row being moved to the new partition. Repro:

CREATE TABLE t (
  id BIGINT,
  val VARCHAR,
  region VARCHAR
) PARTITIONED BY identity(region);

INSERT id=1, region=north;

UPDATE id=1 SET region=east;

After the update,

id=1

appears twice: one row with

region=north

(old/stale) and one with

region=east

(new). From what I understand, this happens because the equality delete only uses the primary key and doesn’t include the partition column, so the delete doesn’t remove the row from the old partition. Has anyone solved this for partitioned tables? Is scheduled compaction the only practical workaround, or is there a write-side fix? Also, does

flink-cdc-pipeline-connector-iceberg

expose any configuration to set the equality fields? Any pointers would be appreciated.

@Pedro Mázala If I wanted to make this change in the Flink CDC main branch itself, will it be accepted? Can I raise a PR for that? I came across these issues as well: github.com/apache/iceberg/issues/11573 github.com/apache/iceberg/issues/7618 Looks like this has been requested by quite a few other people too. I think it might be a relatively simple change, just sending a delete event for the old partition before sending the insert for the new partition. Can we add this to Flink Iceberg connector, or would this have any other impacts that I’m missing? cc: @Leonard Xu

I think opening the PR straight away would not be the best path. Usually changes go through those steps: 1. Mail list discussion about the change 2. Opening either a FLIP or a Jira ticket 3. Asking "blessing" of a PMC member to review it and such 4. And then developing. This is what I'd suggest. But in the end of the day I'm not a PMC myself, so my flow could have gaps 😄