Hello guys I am using the connector

postgres-cdc

but

confirmed_flush_lsn

is not increasing even if I update data on the table and the update is received on Flink. This is causing full storage on Postgres. This is my table on Postgres

CREATE TABLE IF NOT EXISTS aerial_core_development.thing_meta
(
    id integer NOT NULL DEFAULT nextval('aerial_core_development.thing_meta_id_seq'::regclass),
    thing_serial character varying(64) COLLATE pg_catalog."default" NOT NULL,
    floor integer,
    location character varying(32) COLLATE pg_catalog."default",
    location_custom text COLLATE pg_catalog."default",
    name character varying(64) COLLATE pg_catalog."default",
    device_type character varying(64) COLLATE pg_catalog."default",
    timezone character varying(64) COLLATE pg_catalog."default",
    CONSTRAINT thing_meta_pkey PRIMARY KEY (id),
    CONSTRAINT thing_meta_thing_serial_fkey FOREIGN KEY (thing_serial)
        REFERENCES aerial_core_development.thing (serial) MATCH SIMPLE
        ON UPDATE NO ACTION
        ON DELETE CASCADE
)
WITH (
    OIDS = FALSE
)
TABLESPACE pg_default;

ALTER TABLE IF EXISTS aerial_core_development.thing_meta
    OWNER to aerial;

REVOKE ALL ON TABLE aerial_core_development.thing_meta FROM flink;

GRANT ALL ON TABLE aerial_core_development.thing_meta TO aerial;

GRANT DELETE, INSERT, UPDATE, SELECT ON TABLE aerial_core_development.thing_meta TO aerial_core;

GRANT SELECT ON TABLE aerial_core_development.thing_meta TO flink;

This is how I use the connector

CREATE TABLE thing_meta (
        thing_serial VARCHAR NOT NULL,
        timezone VARCHAR
    ) WITH (
        'connector' = 'postgres-cdc',
        'hostname' = 'localhost',
        'port' = '54321',
        'username' = 'flink_bridge',
        'password' = <password>,
        'database-name' = 'aerial_core_development',
        'schema-name' = 'aerial_core_development',
        'table-name' = 'thing_meta',
        'decoding.plugin.name' = 'pgoutput',
        'slot.name' = 'thing_meta_adrian'
    )

Postgres engine version is 10.21

SELECT 
	*, 
	pg_current_wal_lsn(),
	pg_size_pretty(pg_wal_lsn_diff(pg_current_wal_lsn(),restart_lsn)) AS replicationSlotLag
FROM pg_replication_slots;

Do you have any idea why

confirmed_flush_lsn

is not advancing ? Thanks

If there are not a lot of updates in the table you should enable heartbeats for

confirmed_flush_lsn

to progress.

Hi Yaroslav, thanks for taking the time to answer me. I guess heartbeat is enable

io.debezium.connector.common.BaseSourceTask                  [] -    <http://heartbeat.interval.ms|heartbeat.interval.ms> = 300000

Hard to say. The current version of

postgres-cdc

is just a thin wrapper on top of the Debezium connector, so you might want to search / ask in the Debezium community.

I think I fixed by setting

'execution.checkpointing.interval': '1min'

I don't understand why

confirmed_flush_lsn

doesn't increase if we don't set that parameter.

Ah, right, it makes sense that it only flushes during the checkpointing. Same for Kafka offsets, same for writing files with a file sink, etc.