Hi, I’m trying to understand how the cbcopy utility internally handles a data copy operation. I have a few questions regarding its internal copy mechanism: 1. Batch size / chunking: When cbcopy copies data, does it process the data/files in batches or chunks? If yes, how is the batch/chunk size determined? 2. Configurable batch size: Is the batch size or chunk size configurable in cbcopy? If so, which option/parameter can be used to configure it? 3. Internal workflow: What is the internal workflow of cbcopy during a copy operation? For example, does it enumerate the source files first, divide them into batches, read the data into buffers, transfer/write the data, and then verify the result? 4. Commands / operations: What underlying commands, API calls, or storage operations does cbcopy perform while copying the data? I would like to understand what actually happens internally between the source and destination. 5. Parallelism: Does cbcopy use multiple threads/workers or parallel operations while copying? If yes, how is the number of parallel operations determined, and is it configurable? 6. Documentation / logs: Is there any official documentation, debug/verbose logging, or technical reference that explains the internal behavior of the cbcopy utility in more detail? Basically, I’m looking for a technical understanding of how cbcopy actually processes and transfers the data internally, especially regarding batching/chunking, buffering, parallelism, and the underlying operations/commands. Thanks!

Hi... one framing point first, because it answers most of your questions at once: cbcopy isn't a file-copy tool. No file enumeration, no staging to disk, no chunk-then-transfer loop. It's a streaming database-to-database pipe built on the database's own COPY machinery. 1 & 2. Batching and batch size Data isn't chunked, so there's no batch size to configure. The unit of work is the table — or, for partitioned tables, the individual child partition, and once it starts it streams until done. Buffering is whatever the OS pipe and TCP socket provide. Batching does apply to metadata: that part derives from gpbackup and fetches catalog data in batches rather than a few rows at a time, which is where its speed advantage over

pg_dump

comes from. The knobs that actually shape throughput: | Flag | Default | Controls | |---|---|---| |

--copy-jobs

| 4 | Tables copied concurrently | |

--on-segment-threshold

| 1000000 | Row count above which segment-parallel copy kicks in | |

--compression

/

--compress-type

| off / zstd | Wire compression | |

--metadata-jobs

| 2 | Concurrent metadata restore tasks | |

--validate

| true | Row-count check after each table | 3 & 4. Workflow and underlying commands Per table: pick a strategy from

pg_class.reltuples

and cluster topology → start

cbcopy_helper

listeners on the receiving side using

--data-port-range

→

COPY ... TO PROGRAM 'cbcopy_helper'

on source and

COPY ... FROM PROGRAM 'cbcopy_helper'

on destination → helpers stream over direct TCP → compare COPY TO and COPY FROM row counts. No storage APIs, no object store, no intermediate files. Just SQL COPY with

TO PROGRAM

/

FROM PROGRAM

, the

ON SEGMENT

clause for the segment-parallel strategy, and the external table mechanism for the redistribution strategy. Three strategies: - Copy On Coordinator — below

--on-segment-threshold

. Everything funnels through the two coordinators. Single stream. - Copy On Segment — above the threshold, same version and same segment count on both sides. Segments stream directly to their counterparts, no redistribution. - Copy On External Table — everything else. Parallel across segments, with redistribution. 5. Parallelism Two levels. Across tables:

--copy-jobs

. Within a table: determined by topology, not configurable — the two segment strategies fan out across all segments, Copy On Coordinator is single-stream regardless of

--copy-jobs

. So if one large table is slow, check which strategy it landed in before raising

--copy-jobs

. 6. Docs and logs Official docs: cloudberry.apache.org/docs/sys-admin/migration-and-upgrade The README at github.com/cloudberry-contrib/cbcopy goes deeper on strategies and failure signatures. Use

--debug

/ `--verbose`; logs land in

~/gpAdminLogs/cbcopy_$timestamp.log

with succeed/failed table lists alongside. One tie-back to your earlier thread:

--validate

defaults to true, and that row-count comparison is exactly what produced your "Copy to affected rows" failure. Happy to go deeper on any of it. ~tushar Synx Data Labs

@Tushar Pednekar Thank you so much