Hi everyone I’m currently testing Apache Flink CDC

3.7.0-SNAPSHOT

using the Pipeline connector with: • Apache Flink

1.20

• Source: PostgreSQL • Sink: Apache Iceberg

1.9.1

on Amazon Web Services I’m seeing a few issues and wanted to check whether these are known limitations / known bugs, or if fixes are already planned / on the roadmap. ------- New tables are not discovered after restart from savepoint/checkpoint The pipeline discovers new tables correctly during normal runtime. However, after stopping the job and restarting from a savepoint/checkpoint, newly added matching tables are not discovered anymore. Question: Is table discovery state intentionally frozen/restored from the savepoint, or is this a known issue? ------- Schema reconciliation fails after restart if schema changed while job was stopped Scenario: 1. Stop job 2. Apply schema changes to source table 3. Restart job from savepoint The job fails during startup / schema reconciliation instead of evolving the schema and continuing. Question: Is this expected behavior currently, or should schema evolution handle this case? ------- Column rename behavior with Iceberg sink When a column is renamed in PostgreSQL: • Iceberg receives it as a new column (which I understand may be expected in lenient/evolve modes) • old data remains under the old field / does not populate the new renamed field as expected. I observed two scenarios: Scenario 1 (Pipeline running continuously) 1. Column is renamed in PostgreSQL 2. New column is added to sink (Iceberg) with schema change enabled 3. Old data remains under old column, new column receives only new data 4. Old field is removed manually from sink table 5. Subsequent DML operations do not populate the new field; it remains

NULL

Scenario 2 (Pipeline restarted after step 3) 1. Column is renamed in PostgreSQL 2. New column is added to sink (Iceberg) with schema change enabled 3. Old data remains under old column, new column receives only new data 4. Old field is removed manually from sink table 5. After restarting the pipeline, subsequent DML operations populate the new field correctly Questions: 1. Is rename currently treated only as add-column semantics for Iceberg sink? 2. Is there an internal field mapping cache/state that only refreshes after restart? 3. Is proper rename mapping / seamless evolution planned? ------- Would appreciate any guidance on whether these are: • known/open issues • expected current limitations • already fixed in newer snapshots • planned for future releases Thanks

Hi, I'm not very familiar with the Postgres pipeline source behavior, so I'll focus on the Iceberg sink part. The column rename behavior you observed is expected under the LENIENT mode, which is the default schema.change.behavior in Flink CDC. In LENIENT mode, RenameColumnEvent is rewritten by the framework before reaching the sink — it becomes an AddColumnEvent (new column) + AlterColumnTypeEvent (old column set to nullable). The old column is not dropped and no data migrates. This is by design: LENIENT only allows additive, non-destructive changes. The Iceberg sink itself does support true rename. Iceberg's renameColumn is a metadata-only operation that keeps the same fieldId, so existing data maps to the new name automatically. To enable this, set:

pipeline:
  schema.change.behavior: EVOLVE

In EVOLVE mode, the RenameColumnEvent passes through directly to the Iceberg sink, resulting in a seamless, lossless rename. This also explains why your Scenario 2 (restart) works correctly — the writer's cached schema is rebuilt from the current Iceberg table schema after restart, so the new column gets populated.