#2210 Logical replication between two Cloudberry clusters: what the server needs to provide New discussion created by my-ship-it The problem PostgreSQL logical replication reassembles a transaction from one WAL stream. On Cloudberry a transaction is not in one stream. A distributed transaction leaves a piece of WAL on each participating segment and a commit record on the coordinator, and nothing in the WAL ties those pieces together. That is fine for the server itself — two-phase commit already knows who the participants are, in memory, while the transaction is alive. It is not fine for anything reading the WAL afterwards. The coordinator's record says which transaction committed. It does not say where its data went. A receiver that wants to apply distributed transactions to another cluster has to answer two questions that a single-node receiver never asks: 1. Which segments does this transaction have parts on? Without an answer it must wait for every segment in the cluster — and a segment that was never dispatched to will never speak, so the wait never ends. 2. In what order did the transactions commit? Each segment's stream is ordered within itself, but there is no order between segments. Applying the three streams independently can run a later

DELETE

before the earlier

INSERT

of the same row. Both answers exist on the coordinator and nowhere else. What it is for: disaster recovery between clusters that are not the same size The reason I care about this is not "replication" in the abstract. It is that today a Cloudberry cluster can only be stood behind by a copy of itself. Mirrors protect a segment, not a site. For a second site, the choices are a backup and restore, or a second cluster kept in step by physical means — and physical means require the two clusters to have the same shape: the same number of primary segments, segment for segment. That is a hard thing to promise over the life of a system. The DR site is often smaller than production, on purpose; production gets expanded and the DR site does not, or not at the same time; a cluster is moved to different hardware with a different segment count. Any of those ends the arrangement. Logical replication does not have that constraint, because it ships rows and commit order, not blocks and segment positions. The receiver re-routes every row through the target's own distribution policy, so: • source and target segment counts may differ — verified both ways: 3 → 2 and 2 → 3, as well as 3 → 3; • the target is an ordinary cluster the whole time — queryable, not a replay-only standby waiting to be promoted; • a subset works: replicate the tables that matter to the DR site rather than the whole instance; • the two sides may be expanded independently, since neither side's segment count is part of the contract between them. That is the capability I am after. And the target does not have to be a cluster Nothing in the server changes knows what the other end is. They add two things — a list on a WAL record, and an optional output-plugin callback — and an output plugin is the standard way a third party gets at a logical stream. What it does with the stream afterwards is none of the server's business. That matters because "another Cloudberry cluster" is only one of the things people want to feed: • A message bus. A CDC pipeline into Kafka or Pulsar wants the same two answers this proposal supplies, for the same reason. Reading the segment streams independently — which is all a tool can do today — produces rows with no transaction boundary across segments and no order between them, so a downstream consumer sees a later

DELETE

before the earlier

INSERT

of the same row and has no way to know the two belong to one transaction. • Object storage / a lake table. Appending change files to S3-compatible storage, Iceberg, Hudi and so on needs to know when a distributed transaction is complete before it commits a snapshot. "Every segment the coordinator named has delivered its part" is exactly that condition, and it is not derivable from the segment streams alone. • Another database entirely, or an in-house consumer — anything that already speaks the logical replication protocol. None of those need my receiver, and none of them need a change to the server beyond what is here. They need an output plugin of their own, which is a supported extension point, plus the two answers the plugin can now be given. This is also why commit 6 exists in the shape it does: a cluster-wide slot is the same slot on every node, made all-or-nothing, and every such tool needs one. The server exports the piece that creates a slot and keeps the cluster-wide policy out of itself, so a third party can build that policy rather than wait for the server to grow one opinion about it. So I would rather the discussion treat these as an interface than as a hook for one receiver. If the shape is right, a CDC vendor can use it next week without talking to me; if it is wrong, it is wrong for all of them at once — which is the argument for settling it now rather than after the fact. What I am proposing The coordinator's distributed commit record carries the list of participating segments, and logical decoding hands that list to an output plugin through one new optional callback. Nine commits, ~900 lines, of which only two change an interface: •

XACT_XINFO_HAS_DISTRIB_SEGS

— the list on the record, written only where

wal_level

already admits logical decoding. •

distributed_commit_cb

— an optional output-plugin callback, called in WAL order for

XLOG_XACT_DISTRIBUTED_COMMIT

and

XLOG_XACT_DISTRIBUTED_ONEPHASE

, for the slot's own database only. The other seven are consequences, and three of those are plain bug fixes that stand on their own. The nine commits, rebased on current `main`: https://github.com/my-ship-it/cloudberry/tree/mpp-logicalrep-kernel The nine commits | # | Commit | What it does | Why it is needed | Touches | | - | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------- | | 1 | [Fix the cmockery unit test harness for PG16](<https://github.com/my-ship-it/cloudberry… apache/cloudberry