jordepstein
07/10/2026, 4:45 PMRocMarshal
07/13/2026, 8:16 AMFrancis Altomare
07/13/2026, 9:35 AMRocMarshal
07/18/2026, 2:01 PMPhil Scala
07/20/2026, 8:16 PMFrancis Altomare
07/22/2026, 11:08 AMJaejun Lee
08/03/2026, 4:29 AMDiljeet Singh
08/04/2026, 7:11 PMZhe Wang
08/07/2026, 3:08 PMDries Samyn
08/14/2026, 8:38 AMAndreas Bube
08/17/2026, 10:06 AMGrzegorz Liter
08/18/2026, 9:54 AMLorenzo Nicora
08/24/2026, 10:17 AMTiansu Yu
08/31/2026, 12:51 PMRocMarshal
09/09/2026, 2:16 PMAbhibhavya Gupta
09/13/2026, 7:10 AMvishal.balaraman
09/14/2026, 10:23 AMUPDATE changes the value of a partitioned column, the row ends up duplicated. The old partition retains a stale copy, while the new partition contains the updated row.
The root cause is in `RowDataUtils#convertDataChangeEventToRowData`: every UPDATE is currently collapsed into a single after image row tagged as RowKind.INSERT, so no delete is emitted for the row's original partition. Since Iceberg equality deletes are partition scoped by design, the delete that gets written to the new partition cannot reach the stale row in the old partition. Compaction doesn't help either, since there is no delete marker for the stale row.
This has been filed as FLINK-40640, with a proposed fix in PR #4536: split UPDATE handling into an explicit DELETE (before image) + INSERT (after image) pair, and have IcebergWriter write both. I've also added a regression test that reproduces the duplicate against the current main branch and passes with the change. No other tests are affected.
This has also been running in our production CDC pipelines and has resolved the issue there without any regressions.
One possible argument could be that this should be fixed on the Iceberg side. However, from what I understand, the issue is that Flink CDC is currently sending only an INSERT with the after image when running in upsert mode, which means the original row is never deleted. In that case, Iceberg cannot know that it needs to remove the row from the original partition.
Also, recommending that users avoid having a column that can change as an Iceberg partition column doesn't seem like a practical workaround. There are many common business use cases where a frequently updated column can also be a meaningful partitioning column.
Given the size and nature of the change, I wanted to check here first: does this require a FLIP, or would a Jira + PR + committer review be sufficient, given that this is a bug fix rather than a behavioral/design change?
Also, if anyone is familiar with the Iceberg pipeline connector, could you please take a look at the PR?
Thanks!vishal.balaraman
09/16/2026, 8:18 AMJacky Lau
09/16/2026, 4:40 PMUtkarsh Vashishtha
10/03/2026, 7:10 PMJoao Boto
10/07/2026, 2:05 PMrajendar t
10/08/2026, 1:01 PMvishal.balaraman
10/09/2026, 12:40 PM오찬해
10/11/2026, 11:48 PMpeyed3
10/17/2026, 9:30 AMRocMarshal
10/20/2026, 12:07 PMZakelly
10/21/2026, 2:50 PMUtkarsh Vashishtha
10/23/2026, 10:04 PM1.19) on our production instance. I've filed the issue here issues.apache.org/jira/browse/FLINK-41090 and wanted to commit the fix from our fork to upstream: github.com/apache/flink/pull/29585. Please take a look!