Hi all, Raising this for discussion before pushing further on the linked PR. Problem: In the Flink CDC Iceberg pipeline connector, when an

UPDATE

changes the value of a partitioned column, the row ends up duplicated. The old partition retains a stale copy, while the new partition contains the updated row. The root cause is in `RowDataUtils#convertDataChangeEventToRowData`: every

UPDATE

is currently collapsed into a single after image row tagged as

RowKind.INSERT

, so no delete is emitted for the row's original partition. Since Iceberg equality deletes are partition scoped by design, the delete that gets written to the new partition cannot reach the stale row in the old partition. Compaction doesn't help either, since there is no delete marker for the stale row. This has been filed as FLINK-40640, with a proposed fix in PR #4536: split

UPDATE

handling into an explicit

DELETE

(before image) +

INSERT

(after image) pair, and have

IcebergWriter

write both. I've also added a regression test that reproduces the duplicate against the current main branch and passes with the change. No other tests are affected. This has also been running in our production CDC pipelines and has resolved the issue there without any regressions. One possible argument could be that this should be fixed on the Iceberg side. However, from what I understand, the issue is that Flink CDC is currently sending only an

INSERT

with the after image when running in upsert mode, which means the original row is never deleted. In that case, Iceberg cannot know that it needs to remove the row from the original partition. Also, recommending that users avoid having a column that can change as an Iceberg partition column doesn't seem like a practical workaround. There are many common business use cases where a frequently updated column can also be a meaningful partitioning column. Given the size and nature of the change, I wanted to check here first: does this require a FLIP, or would a Jira + PR + committer review be sufficient, given that this is a bug fix rather than a behavioral/design change? Also, if anyone is familiar with the Iceberg pipeline connector, could you please take a look at the PR? Thanks!