Hi everyone I’m using Apache Flink CDC 3.4.0 (Fl...
# flink-cdc
n
Hi everyone I’m using Apache Flink CDC 3.4.0 (Flink 1.2) with the PostgreSQL source connector to replicate data from PostgreSQL into Apache Iceberg on AWS. I’m trying to use Debezium signaling to trigger ad-hoc/incremental snapshots of specific tables. I configured the source with:
Copy code
--  java --
'debezium.signal.enabled.channels' = 'source',
'debezium.signal.data.collection' = 'public.debezium_signal'
and created the signal table in Postgres:
Copy code
--- signal table ---

CREATE TABLE public.debezium_signal (
  id VARCHAR(42) PRIMARY KEY,
  type VARCHAR(32) NOT NULL,
  data VARCHAR(2048)
);
Then I insert signals like:
Copy code
--- trigger adhoc snapshot --- 
INSERT INTO public.debezium_signal (id, type, data)
VALUES (
  '1',
  'execute-snapshot',
  '{"data-collections": ["public.my_table"], "type": "incremental"}'
);
My understanding is that Flink CDC embeds Debezium and passes
debezium.*
properties through, but Flink CDC 3.x also has its own incremental snapshot framework (split enumerator / chunk snapshotting / checkpoint-aware recovery). So I’m unsure whether Flink CDC 3.4.0 actually supports Debezium signaling semantics for triggering snapshots, or whether these configs are simply ignored / only partially honored. Has anyone successfully used Debezium signaling with Flink CDC PostgreSQL source in 3.4.x? Specifically: 1. Does Flink CDC read/process the signal table? 2. Can
execute-snapshot
trigger Flink CDC incremental snapshots? 3. Are there recommended alternatives for ad-hoc table re-snapshotting without restarting the job? Any guidance would be greatly appreciated.
s
Hi!
whether Flink CDC 3.4.0 actually supports Debezium signaling semantics
It doesn’t. Flink CDC has a completely separate implementation for snapshotting. It only uses Debezium for the live CDC phase.
Are there recommended alternatives for ad-hoc table re-snapshotting without restarting the job?
Not really, the decision to snapshot is fully controlled by the source splits: once the snapshot is done, the information is stored in Flink state.