Hey all, I’m running into an issue. We’re using

flink-cdc-pipeline-connector-iceberg (3.6.0-2.2)

to stream MySQL → Iceberg on Glue. When the partition key column is updated, we’re getting duplicate rows instead of the row being moved to the new partition. Repro:

CREATE TABLE t (
  id BIGINT,
  val VARCHAR,
  region VARCHAR
) PARTITIONED BY identity(region);

INSERT id=1, region=north;

UPDATE id=1 SET region=east;

After the update,

id=1

appears twice: one row with

region=north

(old/stale) and one with

region=east

(new). From what I understand, this happens because the equality delete only uses the primary key and doesn’t include the partition column, so the delete doesn’t remove the row from the old partition. Has anyone solved this for partitioned tables? Is scheduled compaction the only practical workaround, or is there a write-side fix? Also, does

flink-cdc-pipeline-connector-iceberg

expose any configuration to set the equality fields? Any pointers would be appreciated.

I think this is because equality-delete mechanism. For

flink-cdc-pipeline-connector-iceberg 3.6.0-2.2

, an

UPDATE

is converted using only the after-image and emitted to Iceberg as an

INSERT

. The connector does not emit a

DELETE(before)

followed by

INSERT(after)

. The Iceberg writer then runs in upsert mode and uses the table's identifier fields as its equality fields. I'd try to ensure the partitioning happens in a "final" (not changing) column... Another potential solution would be forking/updating the connector to ensure a delete is thrown before the insert. Just like

upsert-kafka

does.