https://cloudberry.apache.org/ logo
Join Slack
Powered by
# general
  • g
    #1958 [Ideas] Add instrumentation and latency metrics to the Anser subsystem New discussion created by leborchuk Description Description Anser (
    gpcontrib/anser/
    ) is a runtime pub/sub facility: producer nodes on the segments publish a bloom filter over a join-build key, the coordinator unions the parts, and consumer nodes on the segments receive it and prune probe rows. See 
    gpcontrib/anser/README.md
    for the architecture before starting. Today we can see what the filter did (
    Rows Removed by Bloom Filter
    in 
    EXPLAIN ANALYZE
    ) but nothing about what it cost. A producer blocks until the coordinator acknowledges its part; a consumer blocks until the payload is delivered. If those waits are expensive, the filter can be a net loss and we currently have no way to tell. Goal Make every wait and every queueing delay in Anser measurable, both per query (in 
    EXPLAIN
    ) and cluster-wide (in shared memory). Definition of done • 
    EXPLAIN (ANALYZE, VERBOSE)
    shows producer and consumer wait count and wait time, including for segment-executed nodes. • 
    SELECT * FROM anser.stats()
    shows queue time, service-cycle latency, and the histograms; 
    anser.stats_reset()
    works. • 
    SELECT * FROM anser.session_stats()
    reflects the current session only. • Regression tests pass and contain no timing-dependent expected values. • 
    README.md
    gains a "Metrics" section: every field, its unit, and where it is measured. • No new LWLock acquisitions on the data path; no per-tuple timing calls. • Core (
    src/
    ) is untouched. What to measure There are two distinct vantage points. Do not mix them up — this is the single most important thing to understand before writing code. | Vantage point | Who waits | Where the wait actually happens | | ---------------------------------------- | ---------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | Executor node (segment or coordinator) | the query | segment: the libpq round trip in AnserClientPublish / AnserClientConsumeWait (src/anserclient.c); coordinator: the direct calls AnserPublish / AnserWaitProducersRegistered + AnserWaitReady + AnserConsumeReady (src/anser.c) | | Coordinator internals | the QD backend serving a segment, and the two background workers | AnserWaitSubmissionAck (src/anser.c:906), AnserWaitSlotResult (src/anser.c:1110), AnserGatherServiceCycle (src/anser.c:1235), AnserSendServiceCycle (src/anser.c:1433) | The first belongs in 
    EXPLAIN
    . The second belongs in shared-memory counters, because no single backend sees it. Implementation Suggest three changes in this order. Each is independently reviewable and testable. Change 1 — per-node wait time in EXPLAIN • Time the publish call in the producer node and the receive call in the consumer node. Record, per node: number of waits, total wait time, and longest single wait. • Producer: 
    AnserProducePublishPart()
    in 
    src/anserbloomproduce.c
    — it picks the transport by role, so wrapping it covers both segment and coordinator. • Consumer: 
    ExecAnserBloomFilterConsume()
    in 
    src/anserbloomconsume.c
    . • Store the accumulators in 
    AnserBloomProduceScanState
    / 
    AnserBloomConsumeScanState
    (
    src/anserplanexec.c:81
    and 
    :95
    ). • Print them from 
    anser_produce_explain()
    (
    src/anserplanexec.c:422
    ) and 
    anser_consume_explain()
    (
    :636
    ), gated on 
    es->analyze
    (there is nothing to print without execution) and shown only at 
    EXPLAIN (ANALYZE, VERBOSE)
    — i.e. 
    es->analyze && es->verbose
    . Use 
    ExplainPropertyFloat("Anser Wait Time", "ms", ...)
    and 
    ExplainPropertyInteger("Anser Waits", NULL, ...)
    . The trap you must handle. A field you add to the node state on a segment does not reach the QD. Read the comment at `src/anserplanexec.c:583-588`: the existing code deliberately uses 
    InstrCountFiltered1/2
    because only the fixed fields of 
    CdbExplain_StatInst
    (
    src/backend/commands/explain_gp.c:44
    ) travel back. Your new timers are not in that struct. The supported escape hatch is the per-node extra text channel, which is how 
    Hash
    reports `Extra Text: (seg2) Hash chain length ...`: • 
    PlanState.cdbexplainbuf
    and 
    PlanState.cdbexplainfun
    (
    src/include/nodes/execnodes.h:1152-1153
    ) • collected on the segment by 
    cdbexplain_collectExtraText()
    (
    src/backend/commands/explain_gp.c:1308
    ) and shipped to the QD • copy the pattern from 
    src/backend/executor/nodeRuntimeFilter.c:174-177
     — it is the closest existing analogue (a runtime-filter node doing exactly this) So: allocate 
    cdbexplainbuf
    in 
    *_begin
    when 
    estate->es_instrument
    is set, install a 
    cdbexplainfun
    that appends your numbers, and the QD will show them as 
    Extra Text
    per segment. No core change is needed — if you find yourself editing anything under 
    src/
    , stop and re-read this paragraph. Change 2 — subsystem-wide counters and latency histograms in shared memory • Add a stats block to Anser's shared memory. Either extend 
    AnserControl
    (
    include/anser.h
    ) or add a separate struct; account for it in 
    AnserShmemSize()
    and initialize it in 
    AnserShmemInit()
    (
    src/anser.c
    ), exactly as the existing structures do. • Use 
    pg_atomic_uint64
    (
    src/include/storage/atomics.h
    ) for every counter and bucket, and update them without taking 
    AnserChannelLock
    / 
    AnserRingLock
    . These are on the data path; adding lock traffic for metrics would change what we are trying to measure. • Measure, at minimum: | Metric | Where | | ------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------ | | time a part waits in the submission queue (enqueue → gather picks it up) | stamp the time in AnserEnqueueSubmission() (src/anser.c:860), read it in AnserGatherServiceCycle() (:1235) — add a field to AnserSubmissionEntry | | time a consumer slot waits for delivery (register → send stamps payload) | stamp in AnserRegisterWaitSlot() (:1077), read in AnserSendServiceCycle() (:1433) — add a field to AnserWaitSlot | | producer ack wait, on the QD side | AnserWaitSubmissionAck() (:906) | | consumer delivery wait, on the QD side | AnserWaitSlotResult() (:1110) | | background worker cycle duration, per service | around the AnserGatherServiceCycle() / AnserSendServiceCycle() calls in AnserServiceLoop() (src/anserservice.c) | | counts: parts submitted / accepted / rejected, channels cancelled by timeout, consumers delivered / cancelled | the same call sites … apache/cloudberry
  • g
    #1959 [Ideas] Bloom filter performance: four experiments New discussion created by leborchuk Description This is a research issue. The deliverable for each task is a short report with numbers and a recommendation — not a patch. Code is welcome as evidence, but "we measured X, therefore do Y" is the output we want. A negative result is a successful outcome and must be reported with the same rigour as a positive one: "we tried this, it gained 0.8%, not worth the complexity" is valuable and closes the question permanently. Each of the four tasks below is independent and can be a separate PR/report. Read "How to produce results we can act on" before starting any of them. The current implementation, in facts | Aspect | Anser today | Where | | ------------------ | ---------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------- | | sizing target | 2 bytes (16 bits) per key | bloom_create(), src/backend/lib/bloomfilter.c:96-101 | | bitset floor / cap | 1 MB floor, 64 MB cap | same, plus ANSER_RF_MIN_BYTES / ANSER_RF_MAX_BYTES in gpcontrib/anser/src/anserplan.c:50-51 | | bitset rounded | down to a power of two | my_bloom_power() | | hash functions | optimal_k() = round(0.693 × m/n), capped at 10 | bloomfilter.c:MAX_HASH_FUNCS, optimal_k() | | per-element cost | one hash_any_extended() + k cheap derivations + k random bitset accesses | k_hashes() | | union of parts | for (i...) abits[i] \|= pbits[i]; — one byte at a time | AnserBloomFoldPartInPlace(), gpcontrib/anser/src/anserfilter.c | | union lock | runs while holding AnserChannelLock EXCLUSIVE | AnserStorePayloadDSM(), gpcontrib/anser/src/anser.c:1868 (the Assert) | | wire format | raw header + raw bitset, uncompressed, binary bytea | AnserBloomSerializePart(), sent by anserclient.c | | give up early | never | — | Cloudberry already answers three of these questions differently The core runtime filter (
    src/backend/executor/nodeRuntimeFilter.c
    ) uses 
    bloom_create_aggresive()
    (
    bloomfilter.c
    ), and its choices are the opposite of Anser's: | Anser (bloom_create) | core RF (bloom_create_aggresive) | | | -------------------- | -------------------------------- | ---------------------------------------------------------------------------------------------------------- | | bits per key | 16 | 9 (total_elems * 9 / 8 bytes) | | floor / cap | 1 MB / 64 MB | 128 KB / 2 MB | | hash functions | up to 10 (optimal_k) | 2 or 3, hard-coded, "balance false positive rate and calculation costs" | | refuses to build | never | when bits_per_elem < 1.6 → returns NULL, "too many elements" | | abandons mid-build | never | yes: inner_threshold = bloom_total_bits(bf) / 1.6, then build_suspend = true (nodeRuntimeFilter.c:246,311) | Start every task by explaining why Anser differs from this, and treat the aggressive profile as a candidate answer that someone in this codebase already committed to. Do not invent new thresholds before measuring against that one. How to produce results we can act on This section is the actual hard part of this issue. Results that don't follow it will be sent back, not because of process for its own sake, but because we cannot make a shipping decision from them. 1. Build discipline (most common way to waste a week) • Never benchmark a build configured with 
    --enable-cassert
    or     
    --enable-debug
    only.     Assertions make PostgreSQL 20-40% slower and distort exactly the kind of tight loop you are measuring. Configure a separate optimized build (
    -O2
    , no cassert) and paste the 
    configure
     line into your report. • Record the compiler and version. Auto-vectorization differs between GCC and Clang and between versions; a result without a compiler version is not reproducible. • Pin a baseline commit hash and re-measure the baseline in the same session as the variant. Never compare against numbers from yesterday's machine state. 2. Measure at two layers, always both | Layer | How | Answers | | ------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ---------------------------- | | Micro | a SQL-callable C benchmark added to gpcontrib/anser/src/anser_test.c (follow the existing anser_test_bloom_* helpers), e.g. anser_test_bench_fold(bytes, iters) returning elapsed ms | "is the OR faster?" | | End-to-end | query wall time on a demo/real cluster, TPC-DS with SF=100 | "does the query get faster?" | A micro-benchmark win that does not move end-to-end time is a negative result — report it as such. That happens constantly and is fine. Keep benchmark helpers out of the regression expected output; they are nondeterministic. They are for manual runs. Notes about TPC-DS Scale factor 100 minimum. SF 1 and SF 10 are not acceptable evidence: the tables fit in memory, the filter is pure overhead, and the result tells us nothing about production. Use SF 1000 if the hardware allows, and state the cluster shape (hosts × segments per host). You could use special extension to run TPC-DS https://github.com/avamingli/pg_tpcds Step 1 — find out which queries Anser even touches. The plan pass only injects into one join shape, so most of the 99 queries will be untouched and their timings are noise. Establish the injected subset first: Report that list — it is a result in its own right, and if it is empty, stop and open a bug instead of benchmarking. Expect the star-schema fact/dimension joins (q3, q7, q13, q17, q19, q25, q29, q42, q43, q46, q52, q55, q64, q68, q72, q79, q88, q96, q98 are the usual candidates), but use your measured list, not this one. Step 2 — run the full suite, 3 iterations, `anser.runtime_fi on, everything else identical. Report per-query medians. Step 3 — report in three parts: 1. Injected queries — the detail table: | Query | baseline (s) | variant (s) | Δ% | consumer wait (ms) | | --- | --- | --- | --- | --- | --- | | q19 | | | | | | 1. Non-injected queries — one line: worst regression observed, of que… apache/cloudberry
  • g
    #1962 [Ideas] Adaptive join: correct a wrong Broadcast decision at runtime New discussion created by leborchuk Description The planner picks 
    Broadcast Motion
    for the inner side when it estimates the inner side is small. When that estimate is wrong by 10× or 100×, every segment receives all inner rows and builds the entire hash table — N× the network traffic, N× the build CPU, N× the memory, and often a spill to disk that redistributing would have avoided. The query does not fail; it just runs far slower than the alternative plan, and today nothing notices. Anser could be used to switch from broadcast to redistribute (or vice versa - to discuss). The user-facing requirement, which drives the whole design: boundary must be a region where one algorithm is unambiguously better. Not "they are within 10% and the trend favours one" — a zone where obviously right, with a deliberate no-switch band around it. What we can reuse Anser's channel payload is opaque bytes; only the fold is bloom-specific (
    AnserBloomFoldPartInPlace()
    checks the 
    ABF1
    magic in 
    gpcontrib/anser/src/anserfilter.c
    ). Statistics fold just as easily: | Payload | Fold operation | Use | | ---------------------- | ----------------------- | ----------------------- | | bloom filter (today) | bitwise OR | pruning | | row count / byte count | integer addition | this issue, milestone 1 | | Count-Min sketch | element-wise addition | skew (milestone 3) | | Space-Saving top-K | mergeable summary union | skew (milestone 3) | So milestone 1 needs a second payload type with its own magic a not a new transport. Design that generalization deliberately — a payload-type tag in the header plus a fold callback — because milestone 3 de Milestone 0 — feasibility and correctness (do this first, do not skip) Before any code or benchmarks, answer this in a design note. It is entirely possible that the honest answer to part of this issue is "not feasible without planner changes", and finding that out in week 1 is a success. Facts to start from: • 
    MOTIONTYPE_HASH
    vs 
    MOTIONTYPE_BROADCAST
    (
    src/include/nodes/plannodes.h:1880-1886
    ) differ, at the sender, only in how each tuple's destination is chosen. The receiving slice is the same set of processes either way, and the gang is already running. So flipping the sender's routing is not obviously impossible. • But the join above the motion is only correct if both inputs are placed compatibly. Broadcast-inner works with the outer left wherever it is. Redistribute-inner requires the outer to also be hashed on the join key. If the plan broadcast the inner precisely so the outer would not need a motion, you cannot flip one side in isolation — you would silently produce wrong results, which is far worse than being slow. That yields one case where a runtime switch is provably safe:
    The outer side is already distributed on the join key (it was scanned from
    a table distributed by that key, or a motion earlier in the p
    it there). Then broadcast-inner and redistribute-inner are both correct, and
    the choice is purely about cost.
    Milestone 0 deliverable — a note answering: 1. In which plan shapes is the outer side already hash-distributed on the join key? How often does that happen in TPC-DS at SF100? (Count it — this bounds the value of the whole issue.) 2. Can all senders in a slice agree on a routing switch, and what synchronizes them? (They must agree, or tuples for the same key land on different segments and rows are lost.) 3. What happens to inner tuples already sent before the switch? Does the inner side need re-scanning, and if so what does that cost? 4. Where would the decision be taken and enforced — in 
    nodeMotion.c
    at the sender, or by choosing between two pre-planned alternatives at slice start? 5. For the general (unsafe) case, which is more realistic: planning both alternatives and choosing at execution start, or aborting and re-dispatching with corrected cardinality (what Spark AQE does at shuffle boundaries)? Do not begin milestone 2 until a reviewer has agreed with this Milestone 1 — the cost model and where the boundary is (the core research) What the system already believes The Postgres-planner motion cost (
    cdbpath_cost_motion()
    , 
    src/backend/cdb/cdbpath.c
    ): cost_per_row = (gp_motion_cost_per_row > 0) ? gp_motion_cost_per_row : 2 * cpu_tuple_cost; /* = 0.02 */ motioncost = cost_per_row * 0.5 * (sendrows + recvrows); With R inner rows and N segments: redistribute has 
    recvrows ≈ R
    , broadcast has 
    recvrows = R × N
    . So the model says broadcast-inner beats redistributing both sides when, roughly:
    Copy code
    0.5 · R_in · (1 + N)   <   R_in + R_out
  ⇒  R_out  >  R_in · (N − 1) / 2
    i.e. at N=3 the outer must merely exceed the inner; at N=48 it must exceed it 23×. ORCA carries a separate, blunter rule: 
    optimizer_penalize_broadcast_threshold
    = 100 000 rows by default (
    guc_gp.c:4531
    ). Your first job is to find out whether this linear model is true. It has no term for any of the following, and at least two of them are non-linear. The four cost components — measure each | # | Component | Broadcast | Redistribute | Why it may dominate | | - | ------------------------------ | ---------------------------------------- | --------------- | -------------------------------------------- | | 1 | bytes on the wire | R × N received | R received | linear, but interconnect flow control is not | | 2 | receive-side CPU (deserialize) | R × N | R | linear | | 3 | hash table build | every segment builds all R rows | each builds R/N | N× CPU and N× memory | | 4 | spill to disk | R × width vs work_mem on every segment | (R/N) × width | step function, 2-10× | The strongest candidate for an unambiguous boundary Component 4 is a step function, and steps are exactly what "no doubt" looks like:
    If broadcasting makes the hash table exceed 
    work_mem
    and spill, while
    redistributing keeps it in memory, redistribute wins by a large, verifiable
    margin — regardless of the row count.
    That reframes the trigger from "how many rows" to "which side of the spill boundary each alternative lands on", with the row count as the input to that test. ORCA's flat 100 000 rows is a crude proxy for the same thing that ignores row width, 
    work_mem
    , and segment count. Verify this before building on it. Measure query time for broadcast and redistribute while sweeping inner rows across the point where broadcast starts spilling, at fixed 
    work_mem
    . If the curve shows a sharp knee, that is your boundary and you can defend it. If it degrades smoothly, say so — the recommendation then has to come from the K-factor rule below alone. The switch rule you must propose Express it as a rule with a deliberate dead zone, and state every constant with the measurement that produced it:
    Copy code
    switch broadcast → redistribute  iff
      cost_broadcast  >  K × cost_redistribute        (K ≥ 2, measured)
  AND estimated absolute saving > S                   (e.g. > 1 s or > 1 GB moved)
  AND cost_of_switching_now < estimated saving
    • K is the no-doubt factor. Within K× we do not switch, on purpose. Justify the value you pick from the spread of your own measurements: K must be larger than your measurement noise by a comfortable margin. • S prevents churn on queries too small for any of this to matter. • The switch itself is not free — item 3 of the milestone-0 note. If the inner side must be re-scanned, the switch cost is a full inne… apache/cloudberry
  • g
    #1963 [Ideas] Implement Robust Predicate Transfer (RPT+) on top of Anser New discussion created by leborchuk Description The references, precisely | What | Where | Note | | ----------------------- | -------------------------------------------------------------------------------------------- | --------------------------------------------- | | RPT paper | Junyi Zhao et. all Debunking the Myth of Join Ordering: Toward Robust SQL Analytics | https://arxiv.org/pdf/2502.15181 | | DuckDB RPT extension | https://github.com/robust-sql/robust | this is the baseline RPT | | RPT+ paper | Yiming Qiao, Peter Boncz, Huanchen Zhang, Robust Predicate Transfer with Dynamic Execution | https://vldb.org/pvldb/vol19/p1278-qiao.pdf | | DuckDB RPT+ extension | https://github.com/embryo-labs/dynamic-predicate-transfer | this is RPT+ | Read both repos, but know which is which: 
    robust-sql/robust
    implements plain Predicate Transfer (DuckDB's native bloom filter, 12 bits/key; reports 1.76× geomean on JOB), while the paper is specifically about fixing RPT's overhead and regressions. ⚠️ Licensing. Cloudberry is Apache-2.0. Check the license of both repos before reusing even a snippet, and prefer implementing from the paper. If you do reuse anything, it needs attribution plus `LICENSE`/`NOTICE` handling — raise it in the PR rather than deciding alone. What the paper actually proposes RPT (the baseline) makes Yannakakis' algorithm practical: model the query as a join graph 
    G(V,E)
    where edge weight = number of shared join attributes, build a maximum spanning tree rooted at the largest table (
    LargestRoot
    ), then propagate bloom filters over that tree in a forward pass (leaves→root) and a backward pass (root→leaves), fully reducing tables before the join phase. Its weakness: it ignores equivalence relationships between join keys, producing redundant and oversized filters, and pays heavy scan + BF-construction overhead — enough to cause real regressions. RPT+ adds three things: 1. Asymmetric Transfer Plan (ATP) — different tree shapes for the two passes, because they do different jobs. Forward = collect information, so a deep chaining tree is best (filter table i before building the filter for i+1, avoiding oversized BFs). Backward = distribute information, so a wide broadcast tree is best (build the final filter once and reuse it for every table in the same equivalence class). Chains for different equivalence classes are linked through "bridge tables" (the root of each class's induced subtree). 2. Cascade filters — min/max for block-level skipping plus a bloom filter for tuple-level filtering, applied coarse-to-fine. 3. Dynamic pipelines — runtime selectivity monitoring that abandons filter construction when it will not pay off (Algorithm 2 in §5.2). Reported speedups over DuckDB v1.3.0: 1.47× JOB, 1.28× SQLStorm, 1.17× TPC-H, 1.01× Appian, "avoid[ing] the significant performance regressions observed with the original RPT". Constants from §5.5, use these as starting values, do not invent your own: | Parameter | Value | Meaning | | ---------- | ------------------------------- | ----------------------------------------------------- | | γ | 100 K tuples | rows observed before deciding | | τ_sel | 0.35 | abandon if > 35% of tuples survive | | τ_prog | 0.6 | only abandon while < 60% through the scan | | M_avail | 64 GB | memory budget for materialization | | τ_stop | 0.9 | stop probing a filter that rejects too little | | filter | 20 bits/key, 7 hash functions | FPR 6.1×10⁻⁵ | | probe cost | 2.48 cycles/tuple | 64-byte cache-line blocks, 32-bit sectors, VPGATHERDD | Sampling is sequential, not random — random access breaks streaming execution. On JOB, τ_sel in 0.1-0.3 gave the best speedups; τ_sel = 0 (drop almost everything) hurt. Why Anser fits — and the one place it does not | RPT+ concept | Anser today | Gap | | ----------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------- | | equivalence class of join attributes | condition_key — anser.h documents it as exactly "the optimizer-generated equivalence-class symbols" | naming only | | backward pass: build the final filter once, broadcast to every table in the class | one channel per condition, union once at the coordinator, deliver to every registered consumer | already implemented | | combining filters built by different workers | gather service folds parts by bitwise OR | already implemented (the paper is single-node; this is our addition) | | forward pass: chaining, table i reduced before it builds the filter for i+1 | channels are independent, with no ordering between them | the hard part — see below | | dynamic pipeline abandonment (Alg. 2) | never abandons | overlaps Issue 2 task 2.4 | | cascade filter, blocked/sectorized BF | classic bloom: k random bitset accesses | overlaps Issue 2 tasks 2.1/2.3 | | one filter per (table, equivalence class) | plan pass injects one producer+consumer per hash join | needs join-graph analysis in anserplan.c | Anser's backward pass is essentially free — it is what the extension already does. The forward chaining pass is where this project lives or dies, for a reason that does not exist in DuckDB: • In DuckDB, a chain hop is an in-process pipeline breaker: microseconds of coordination. • In Anser, every hop is a global barrier: all segments publish → coordinator unions → all segments receive. One hop costs a full segment→QD→segment rendezvous. JOB queries join 5-10 tables, i.e. 4-9 sequential hops. If one hop costs ~30 ms, a 7-table chain adds ~200 ms of pure latency to a query whose total runtime may be under a second. That arithmetic, with a measured hop latency, decides whether M3 below is worth attempting at all. ⚠️ Correctness rule you must not get wrong. It is tempting to do the forward chaining locally on each segment and avoid the barrier. That is only correct when the tables are co-distributed on the join key. Otherwise segment s holds only part of table _j_'s key set, and filtering table j+1 with it drops row… apache/cloudberry
  • g
    #1967 [DISCUSSION] Making Apache Cloudberry an Agent-Native Analytical Database New discussion created by MisterRaindrop I'd like to discuss a possible direction for Apache Cloudberry: making Cloudberry an open analytical backend for AI agents, rather than building another agent framework inside the database. AI agents such as Codex, Claude, Maka, and other MCP-compatible agents are increasingly becoming a new interface for interacting with data systems. The basic idea is:
    Copy code
    Agent
         Codex / Claude / Maka
            /      |      \
           /       |       \
         Web    Cloudberry  MCP Apps
                   │
            ┌──────┼──────────┐
            ▼      ▼          ▼
           SQL    RAG      Semantic
            │      │
         Iceberg  Lance
    Cloudberry already has an MCP server and a mature MPP analytical engine. Therefore, I don't think Cloudberry needs to build another agent runtime. The external agent can be responsible for: • reasoning and planning • conversation and context • web search • external MCP tools • multi-step tool orchestration Cloudberry can focus on what an analytical database is good at: • SQL analytics • MPP execution • structured data analysis • vector / semantic retrieval • structured + unstructured analysis The responsibility boundary could be:
    Copy code
    Agent:
    decide what to do

Cloudberry:
    decide how to analyze the data efficiently
    Structured + AI-Native Data Cloudberry already provides a strong foundation for structured analytical workloads through SQL, MPP execution, and lakehouse integration. One possible missing piece is an AI-native data layer. A possible architecture is:
    Copy code
    Cloudberry
                     │
              Unified Analytics
                     │
          ┌──────────┴──────────┐
          ▼                     ▼
       Iceberg                 Lance
          │                     │
     Structured          AI-native Data
       Data             Vector / Text /
                        Multimodal Data
    Iceberg can continue to serve structured analytical/lakehouse workloads. Lance could complement it as an external data source optimized for AI-oriented datasets. This could eventually allow queries combining structured business data with semantic or multimodal data. For example:
    Find customers whose revenue dropped by more than 20% in the last three months, then analyze their support tickets to determine the most common complaints.
    Conceptually:
    Copy code
    Iceberg / Cloudberry
        ↓
Structured SQL analysis
        ↓
Declining customers
        ↓
Lance semantic/vector search
        ↓
Support documents
        ↓
Cloudberry aggregation
        ↓
Agent reasoning
    This is closer to analytical RAG than traditional Top-K RAG. Why Lance? The goal is not simply to add another vector index. Lance is interesting because it is designed around AI-oriented datasets containing combinations of:
    Copy code
    metadata
text
vectors
images
audio/video references
multimodal features
    This makes it potentially useful as an AI-native data layer alongside Iceberg. Cloudberry could remain responsible for SQL, JOIN, aggregation, MPP execution, and distributed query planning, while Lance provides storage and retrieval capabilities for AI-oriented datasets. Why Not Just pgvector? Cloudberry already supports pgvector, and pgvector is a good solution for storing and searching vectors inside PostgreSQL-compatible relational tables. I see pgvector and Lance as solving different problems.
    Copy code
    pgvector

Cloudberry native table
        │
        ├── relational columns
        └── vector column
    This is a natural solution when embeddings are part of relational data. The proposed Lance integration targets external AI-native datasets:
    Copy code
    Lance Dataset
     │
     ├── metadata
     ├── text
     ├── vectors
     ├── multimodal data
     └── vector indexes
    Therefore, Lance would not replace pgvector. The three layers could coexist:
    Copy code
    pgvector
    → vectors in native Cloudberry tables

Iceberg
    → structured lakehouse datasets

Lance
    → AI-native / multimodal datasets

Cloudberry
    → unified MPP analytical engine
    A simple way to describe the distinction is:
    pgvector provides vectors inside PostgreSQL. Lance provides an AI-oriented dataset layer. Cloudberry provides the analytical engine across them.
    RAG and Hybrid Retrieval Once Cloudberry can access Lance datasets, we could expose retrieval capabilities through the existing MCP server. For example:
    Copy code
    execute_query()
vector_search()
hybrid_search()
search_documents()
retrieve_context()
    An external agent could then combine these tools. For example:
    Copy code
    Codex / Claude
                    │
                    MCP
                    │
              Cloudberry
              /          \
             /            \
       SQL Analytics     RAG
            │             │
         Iceberg        Lance
    The important point is that Cloudberry does not need to know whether the caller is Codex, Claude, Maka, or another agent. MCP provides the common interface. Analytical RAG A longer-term opportunity is to go beyond traditional RAG. Traditional RAG usually works as:
    Copy code
    Documents
    ↓
Vector Search
    ↓
Top-K
    ↓
LLM
    This works well for retrieval, but not for questions such as:
    What percentage of all customer complaints last year were related to query latency, grouped by month?
    This requires:
    Copy code
    Large document dataset
        ↓
Semantic retrieval/filtering
        ↓
AI extraction/classification
        ↓
Cloudberry MPP
        ↓
COUNT / GROUP BY / JOIN
    This could be an interesting area where Cloudberry's existing analytical engine provides capabilities beyond a standalone vector database. Semantic Layer Another possible future direction is a lightweight semantic layer. Instead of requiring an agent to infer business meaning directly from physical schemas, Cloudberry could expose concepts such as:
    Copy code
    Metrics:
    revenue
    churn_rate

Dimensions:
    region
    customer
    product

Relationships:
    customer → orders
    customer → support_tickets
    The external agent could use this semantic information before generating SQL or retrieval requests. This could improve the reliability of natural-language analytics without requiring Cloudberry itself to implement an agent runtime. Proposed First Version I think the first implementation should remain intentionally small. Phase 1: Read-only Lance FDW The initial goal could simply be:
    Allow Cloudberry to query an existing Lance Dataset as an external table.
    For example: CREATE FOREIGN TABLE lance_documents ( id bigint, customer_id bigint, content text, embedding float4[] ) SERVER lance_server OPTIONS ( uri 's3://bucket/documents.lance' ); Initially support: • schema mapping • sequential scan • projection pushdown • filter pushdown • read-only access No INSERT / UPDATE / DELETE would be required initially. Data could be generated by existing Lance tools or a simple export utility. Phase 2: Vector Top-K Pushdown Then support queries such as: SELECT id, content FROM lance_documents WHERE customer_id IN (...) ORDER BY embedding <-> query_embedding LIMIT 20; Instead of:
    Copy code
    Read all vectors
      ↓
Cloudberry distance calculation
      ↓
Sort
      ↓
LIMIT
    Cloudberry could recognize:
    Copy code
    Vector distance
+
LIMIT
    and push the operation into Lance:
    Copy code
    Cloudberry Planner
        ↓
Vector Top-K Pushdown
        ↓
Lance Vector Index
        ↓
Top-K candidates
    This would be the first step toward a vector-aware analytical engine. Phase 3: Distributed Vector Top-K Cloudberry's MPP architecture could later provide: ``` QD │ ┌──────────┼──────────┐ ▼ ▼ ▼ QE0 QE1 QE2 │ │ │ Lance Lance Lance │ │ │ Top-K Top-K Top-… apache/cloudberry
  • d
    The recording of our latest Apache Cloudberry bi-weekly community meeting is now available:

    youtube.com/watch?v=fmSQyPLb8yI▾

    Highlights from the meeting: • Apache Cloudberry 2.2 release preparation, with 1,000+ commits and around 1,500 changed files • Security fixes and PostgreSQL updates • Query execution progress reporting • Bloom filter pushdown and runtime instrumentation • Data Lake FDW and Iceberg support • PXF 2.2 release preparation and future PXF 3.0 plans • Cloudberry Command Center’s ongoing transfer into the Apache ecosystem • New ecosystem tools and integrations, including CloudDM and BladePipe support • Community event planning and roadmap discussions The next meeting is scheduled for September 18, 2026. Everyone is welcome to watch the recording, review the open topics, and participate in the discussions!
  • d
    🚀 Apache Cloudberry 2.2.0 Preview is now available Apache Cloudberry 2.2.0 is shaping up to be our broadest release yet, with coordinated releases across Cloudberry Core, PXF, Backup, and the first standalone release of Cloudberry Go Libs. Some highlights: • PostgreSQL 14.4 → 14.9, with a new process for continuously tracking upstream minor releases • 41 CVE fixes compared with 2.1.0 • Six new extensions, including 
    diskquota
    , 
    try_convert
    , 
    yezzey
    , `gp_stats_collector`and 
    reject_partition_fullscan
    • Expanded parallel query processing and new interconnect statistics views • Support for Rocky Linux 10 and Ubuntu 24.04 • New PXF observability and control tools, including 
    pxf_stat_activity
    • New Cloudberry Backup tools: 
    gpbackman
    and 
    gpbackup_exporter
    • The first standalone release of 
    apache/cloudberry-go-libs
    Please note that 2.2.0 has not been officially released yet, and the final contents may still change. Read the preview blog post: cloudberry.apache.org/blog/apache-cloudberry-2.2.0-preview Feedback and contributions are welcome as we move toward the final release!
  • g
    #2005 [Proposal] Fix ORCA selectivity damping for predicates with outer references New discussion created by Waloid24 Proposers Andrei Proposal Status Under Discussion Abstract Cloudberry's 
    SelectivityOfPredicate()
    can estimate that adding an outer-reference predicate to a conjunction increases the fraction of rows passing the filter. This proposal applies damping only when combining outer predicates’ conditional selectivity estimates, then multiplies the result by the local selectivity. It reuses the existing conjunction estimator and honors 
    optimizer_damping_factor_filter
    instead of a hardcoded default. Motivation Motivation ORCA's 
    CFilterStatsProcessor::SelectivityOfPredicate()
    estimates predicate selectivity when evaluating candidate index conditions. Its current damping formula can increase the estimated fraction of qualifying rows when another condition is added through 
    AND
    . This contradicts the fact that a conjunction cannot admit more rows than any of its constituent conditions. Current mathematical model Let the predicate be: $$ L \land E_1 \land \cdots \land E_k, $$ where $L$ combines the local conditions and each $E_i$ is a conjunct containing outer references. The function first estimates statistics after applying $L$. Its initial selectivity is: $$ s_L = \frac{N_L}{N}, $$ where $N$ is the estimated base-table cardinality and $N_L$ is the estimated cardinality after local filtering. For each outer predicate, it assigns a selectivity multiplier $q_i$: • For a recognized equality between a local column and an outer expression, $q_i=1/D_i$, provided the estimated NDV $D_i\ge1$. • Otherwise, it uses the default selectivity $0.4$. • Constant 
    TRUE
    expressions are ignored. The NDVs come from the statistics after local filtering. Accordingly, these multipliers can be interpreted as heuristic estimates of the conditional probabilities $P(E_i\mid L)$. The function first multiplies the estimates: $$ r_{\mathrm{raw}} = s_L\prod_{i=1}^{k}q_i. $$ It then counts the outer predicates, adding one for the complete local condition if $s_L<1$: $$ m = k + \mathbf{1}_{s_L<1}. $$ If $m \le 1$, no final damping correction is applied: $$ S_{\mathrm{current}} = r_{\mathrm{raw}}. $$ If $m > 1$, the function divides the product by the damping correction and caps the result at one: $$ S_{\mathrm{current}} = \min\left(1,\frac{r_{\mathrm{raw}}}{0.75^m}\right). $$ The coefficient in this final correction is always 
    0.75
    , taken from 
    PstatsconfDefault()
    , rather than the configured 
    optimizer_damping_factor_filter
    . Required properties Let $p_i=P(A_i)$ be the selectivity of predicate $A_i$, for $n$ predicates evaluated over the same population. Their individual selectivities do not determine their conjunction: the result also depends on how the qualifying rows overlap. Let $F_d(p_1,\ldots,p_n)$ estimate the conjunction at damping coefficient $d\in[0,1]$. Damping should interpolate between independence ($d=1$) and maximum overlap ($d=0$): $$ F_1(p_1,\ldots,p_n)=\prod_i p_i, \qquad F_0(p_1,\ldots,p_n)=\min_i p_i. $$ Maximum overlap means that every row satisfying the most restrictive predicate also satisfies the others. This model covers dependence that increases overlap relative to independence, not every possible dependence. The estimate should satisfy: $$ \prod_i p_i\le F_d(p_1,\ldots,p_n)\le\min_i p_i. $$ The upper bound is required by conjunction semantics; the product lower bound belongs to the chosen damping model. Adding a predicate with selectivity $p_{n+1}$ must not increase the estimate, with the existing inputs and coefficient unchanged: $$ F_d(p_1,\ldots,p_n,p_{n+1})\le F_d(p_1,\ldots,p_n). $$ A neutral predicate with selectivity one must leave the result unchanged, as must reordering the inputs. For example, selectivities $0.1$ and $0.2$ allow an estimate between $0.02$ and $0.1$. Applying this to outer predicates The outer estimates $q_i$ use statistics after local filtering, so we interpret them as approximations to $P(E_i\mid L)$. The probability decomposition is: $$ P(L\cap E_1\cap\cdots\cap E_k) =P(L),P(E_1\cap\cdots\cap E_k\mid L). $$ Therefore, let the proposed overall estimate $S_d$ be: $$ S_d=s_L,F_d(q_1,\ldots,q_k). $$ Damping combines the outer estimates within the locally filtered population. The local selectivity remains a separate multiplier. For $k\ge1$: $$ s_L\prod_i q_i\le S_d\le s_L\min_i q_i\le s_L. $$ Without outer predicates, return $s_L$. With one outer predicate, return $s_Lq_1$ for every coefficient. Theoretical counterexample Suppose local filtering retains an estimated 1,000 of 10,000 rows, so $s_L=0.1$. Add an outer equality with $D_1=1$, giving $q_1=1$. This multiplier excludes no locally qualifying rows, so the combined estimate should remain $0.1$. Instead, the current implementation counts two factors and returns: $$ S_{\mathrm{current}}=\frac{0.1\times1}{0.75^2}\approx0.177778. $$ Adding a conjunct increases the estimated qualifying population from 1,000 to approximately 1,778 rows. Capping the result at one does not prevent this violation. Even a cap at $s_L$ would be insufficient: with $s_L=0.1$ and $q_1=0.2$, the current formula returns approximately $0.035556$, exceeding the conditional-model result $s_Lq_1=0.02$. The change would preserve these bounds and honor the configured damping coefficient in an estimate used for index selection. SQL demonstrations Create a 10,000-row inner table and a single-row outer table. The bitmap indexes and disabled hash joins encourage ORCA to consider index conditions containing outer references. The estimates below refer to the value returned by 
    CFilterStatsProcessor::SelectivityOfPredicate()
    , which can be inspected in a debugger. They are not necessarily the row estimates shown by 
    EXPLAIN
    . SET optimizer = off; CREATE TABLE damping_inner ( id integer, a integer, b integer, c integer, z integer ) USING heap DISTRIBUTED RANDOMLY; INSERT INTO damping_inner SELECT n, n % 10, (n / 10) % 5, (n / 50) % 4, 0 FROM generate_series(0, 9999) AS g(n); CREATE TABLE damping_outer ( b integer, c integer, z integer ) USING heap DISTRIBUTED REPLICATED; INSERT INTO damping_outer VALUES (0, 0, 0); CREATE INDEX damping_inner_abcz ON damping_inner USING bitmap (a, b, c, z); CREATE INDEX damping_inner_bcza ON damping_inner USING bitmap (b, c, z, a); ANALYZE damping_inner; ANALYZE damping_outer; SET optimizer = on; SET optimizer_enable_hashjoin = off; SET optimizer_damping_factor_filter = 0.75; The generated data has the following properties: | Property | Value | | ---------- | ------ | | Inner rows | 10,000 | | NDV of a | 10 | | NDV of b | 5 | | NDV of c | 4 | | NDV of z | 1 | 1. Local predicate only EXPLAIN (ANALYZE, TIMING OFF) SELECT i.* FROM damping_outer AS o CROSS JOIN damping_inner AS i WHERE i.a = 1; The query returns exactly 1,000 rows, corresponding to a selectivity of $0.1$. In the debugger, the function returned `0.099999003112316131`; the small difference comes from ORCA's normalization of floating-point frequencies. This is the baseline. Damping does not affect this single equality. The calculations below use the idealized value $s_L=0.1$ for clarity. 2. Add a neutral outer predicate EXPLAIN (ANALYZE, TIMING OFF) SELECT i.* FROM damping_outer AS o CROSS JOIN damping_inner AS i WHERE i.a = 1 AND i.z = o.z; Both 
    i.z
    and the single outer row's 
    o.z
    are zero, so the additional condition excludes no rows: $$ s_L=0.1,\qquad q_z=1. $$ The actual output remains 1,000 rows. | Implementation | Calculation | Approximate selectivity | | -------------- | ------------ | ----------------------- | | Original | $s_L/0.75^2$ | 0.177778 | | Expected | $s_L\times1$ | 0.1 | The original formula increases the estimate when a conjunct is added. Instead, the corrected formula should preserve the baseline estimate, as required. Implementation Implementation We propose keeping the local selectivity se… apache/cloudberry
  • y
    Hi everyone, I’m testing cbcopy between Greenplum 7.3.3 (source) and Apache Cloudberry 2.1.0 (destination) and I’m seeing two different issues. Issue 1 – Broken pipe after ~2 minutes When using a wide dynamic port range (1024-65535), some partitions consistently fail after around 2m 9–10s with: could not write to COPY program: Broken pipe This makes me suspect a firewall/NAT/load-balancer TCP session timeout, or possibly something on the destination side killing cbcopy_helper. Issue 2 – Narrow port range causes hang / 0% progress When I restrict the helper port range to 40000-40050, cbcopy gets stuck at: Table copied: (0/1) 0% and no worker/COPY/helper connection logs appear after retrieving the partition metadata. So I'm wondering if the 40000-40050 range is being blocked or filtered somewhere in the network. I also tested a small table. With the connection established, data was actually transferred (12 rows written), but cbcopy then marked the table as failed with: [ERROR]:-[Worker 0] Failed to copy table ...: Copy to affected rows 12 are ... Unfortunately, the error is truncated, so I'm investigating whether this is a row-count/validation issue after the copy, rather than a network problem. Questions: 1. Has anyone seen cbcopy/cbcopy_helper connections fail consistently after ~120–130 seconds with Broken pipe? 2. Is there any known firewall/network configuration required for the cbcopy helper's dynamic port range? 3. If using a restricted range like 40000-40050, are there any additional cbcopy/Cloudberry settings required? 4. Could the Copy to affected rows validation error be related to Greenplum → Cloudberry compatibility, or should I treat it as a separate issue? 5. Any recommended way to debug this further (tcpdump, firewall logs, cbcopy debug logs, etc.) would be appreciated. Environment: • Source: Greenplum Database 7.3.3 / PostgreSQL 12.12 • Destination: Apache Cloudberry 2.1.0 / PostgreSQL 14.4 Thanks!
    t
    • 2
    • 5
  • g
    #2021 [Ideas] Anser: Folding a segment's parallel workers stat before the coordinator sees them New discussion created by leborchuk Description Folding a segment's workers before the coordinator sees them Status: idea, not implemented. No wire-protocol change. Original PR with Anser - #1942 The problem A parallel slice runs 
    numsegments × parallel_workers
    processes and every one of them publishes its own part. The coordinator folds them all and then returns the merged payload to every consumer, one blocking write at a time. Both halves scale with the process count rather than the segment count, and the payload does not shrink along the way: a bloom union is a bitwise OR, so each part is a whole bitset. Measured on 4 segments with a 1 MB filter, and extrapolated to 128 segments with a 64 MB one (the ceiling 
    anser.max_info_size
    allows): | Cluster | parallel_workers | Parts folded | Deliveries | Through the coordinator | | -------------- | ---------------- | ------------ | ---------- | ----------------------- | | 4 seg, 1 MB | 1 | 4 | 4 | 9.6 MB | | 4 seg, 1 MB | 4 | 16 | 16 | 38 MB | | 128 seg, 64 MB | 1 | 128 | 128 | 18.7 GiB | | 128 seg, 64 MB | 4 | 512 | 512 | 74.7 GiB | | 128 seg, 64 MB | 8 | 1024 | 1024 | 149.3 GiB | Per join, over one coordinator NIC, single-threaded, with a base64 decode and a CRC per part. At 4 segments and 16 workers the fan-out already took 1140 ms, which overran the old 1 s consumer deadline and left every consumer unfiltered. Raising the deadline (now 100 s by default) makes the filter arrive but does not make it cheap: the first subscriber is served in under a millisecond and the last over a second later. The idea The workers of one segment are processes on one host. They do not need the coordinator to combine their parts with each other — only to combine them with other segments'. So fold locally first, and let one process per segment speak to the coordinator on behalf of the rest.
    Copy code
    now:      16 workers ──16 parts──> QD ──16 pushes──> 16 workers
proposed: 16 workers ──local fold──> 4 leaders ──4 parts──> QD
                                     ──4 pushes──> 4 leaders ──local──> 16 workers
    Effect on the numbers above: parts folded and deliveries both drop by a factor of 
    parallel_workers
    , back to the rows where it equals 1. The 128-segment, 8-worker case falls from 149.3 GiB to 18.7 GiB — the coordinator stops paying for parallelism it gains nothing from. What it does not fix: the remaining 
    numsegments
    parts still cross the coordinator, so a 128-segment cluster still moves ~19 GiB per join for a 64 MB filter. A full fix is an all-reduce over the interconnect, where per-node traffic is ~2 × payload regardless of cluster size. Local folding is the cheaper first step and is a prerequisite for that anyway — the interconnect exchange wants one participant per segment, not one per worker. Why this fits the existing design • The wire protocol does not change. A leader publishes the same 
    ANSER_WIRE_KIND_PART
    message it publishes now, and the coordinator still folds N parts without caring what N is. Only 
    ANSER_RF_PRIV_N_PRODUCERS
    gets a different value: 
    numsegments
    instead of 
    numsegments × parallel_workers
    . • 
    fold()
    already exists     in 
    AnserPayloadOps
    (
    anserpayload.c
    ) and is the same bitwise OR the coordinator uses. The local fold calls the same function on the same payload type; nothing type-specific is duplicated. • The count stays authoritative. A gang is all-or-nothing — 
    cdbgang_createGang_async()
    either returns a full gang or raises — so 
    numsegments
    is exactly the number of leaders that will run, just as 
    numsegments × parallel_workers
    is exactly the number of processes today. What has to be built 1. A per-segment rendezvous. The workers of a slice need somewhere to deposit a part and somewhere to read the merged result. A DSM segment keyed by 
    (gp_session_id, gp_command_count, condition_id)
    is the obvious shape; the QEs of one gang share a host but not a 
    ParallelContext
    , so nothing existing can be reused directly. 2. A local barrier. The leader publishes once its workers have all deposited. This is where the design earns its keep or fails: a worker that never arrives — squelched, or erroring — must not strand the segment. The barrier needs the same fail-open discipline the consumer deadline has, and the leader should publish what it has rather than wait indefinitely. 3. Who is the leader. One process per segment per slice has to be distinguishable. 
    Gp_is_writer
    is not it (all the QEs in the log are writers); this needs an explicit answer. 4. Local redistribution. After the leader receives the merged filter, its workers need it. Same shared segment, in the other direction. Risks • A barrier is a new way to hang. Today a lost part costs one part; with a barrier it can cost a whole segment's progress. Every wait must have a deadline and a fail-open path, and the leader must never block on a worker that has already finished or died. • Shared memory is not transaction-aware. The segment has to be released on abort as well as on completion — the same problem 
    dl_resource.c
    solves for datalake_fdw with a resource-owner callback, and 
    comm/pax_resource.cc
    for PAX. • It only moves the knee. 128 segments still overwhelm a star topology. If the target is large clusters with large filters, this buys time; it is not the destination. Use case/motivation No response Related issues No response Are you willing to submit a PR? • Yes I am willing to submit a PR! apache/cloudberry
  • g
  • g
    Comment on #2024 Ubuntu 24 support status Discussion answered by tuhaihe Hi, we are working on the upcoming 2.2 release. This new version will add support for Ubuntu 24.04. If that sounds good, please be patient and wait 1-2 weeks for it. apache/cloudberry
  • y
    Hi, I’m trying to understand how the cbcopy utility internally handles a data copy operation. I have a few questions regarding its internal copy mechanism: 1. Batch size / chunking: When cbcopy copies data, does it process the data/files in batches or chunks? If yes, how is the batch/chunk size determined? 2. Configurable batch size: Is the batch size or chunk size configurable in cbcopy? If so, which option/parameter can be used to configure it? 3. Internal workflow: What is the internal workflow of cbcopy during a copy operation? For example, does it enumerate the source files first, divide them into batches, read the data into buffers, transfer/write the data, and then verify the result? 4. Commands / operations: What underlying commands, API calls, or storage operations does cbcopy perform while copying the data? I would like to understand what actually happens internally between the source and destination. 5. Parallelism: Does cbcopy use multiple threads/workers or parallel operations while copying? If yes, how is the number of parallel operations determined, and is it configurable? 6. Documentation / logs: Is there any official documentation, debug/verbose logging, or technical reference that explains the internal behavior of the cbcopy utility in more detail? Basically, I’m looking for a technical understanding of how cbcopy actually processes and transfers the data internally, especially regarding batching/chunking, buffering, parallelism, and the underlying operations/commands. Thanks!
    t
    • 2
    • 2
  • g
    #2030 [Proposal] Remove `gpcontrib/gpmapreduce` starting from Cloudberry 3.0 New discussion created by tuhaihe Proposers Dianjin Wang (@Dianjin Wang) Proposal Status Under Discussion Abstract This proposal suggests removing 
    gpmapreduce
    from Apache Cloudberry, starting with the 3.0 development line. Concretely, on 
    main
    (currently 
    3.0.0-devel
    ) we would delete: • the 
    gpcontrib/gpmapreduce/
    directory (44 files, ~13,500 lines), • the 
    gpmapreduce
    client binary that is currently installed into 
    $GPHOME/bin/
    , • the 
    --enable-mapreduce
    configure option and the unused 
    USE_MAPREDUCE
    define, • the 
    gpmapreduce_help
    file shipped by 
    gpMgmt/doc/
    . The 2.x line (
    REL_2_STABLE
    , currently 
    2.2.0-incubating
    ) would not change. 
    gpmapreduce
    keeps building and shipping there for the entire supported life of 2.x, so users who still depend on it have a supported place to stay. The removal takes effect in 3.0 and remains in effect for 3.x. Motivation 1. This functionality is no longer needed according to the commit 
    4d926ede502
     Commit 
    4d926ede502
    (2024-09-19) disabled the gpmapreduce regression test in 
    gpcontrib/Makefile
    , with the following commit message:
    As gpmapreduce is implemented for compatibility of execution of mapreduce,
    which is no longer need now, we disable gpmapreduce regress test in this commit.
    However, only the test was turned off. The build was not. Our release build script still passes 
    --enable-mapreduce
    (
    devops/build/automation/cloudberry/scripts/configure-cloudberry.sh
    ), and 
    gpcontrib/gpmapreduce/Makefile
    still installs the binary into 
    $GPHOME/bin/gpmapreduce
    . 2. Upstream removed it three years ago Greenplum deleted the same code in 
    6bc9129c90b
    (2023-07-13):
    It's decided that this functionality will not be provided for GPDB7 and
    forward. Hence, deleting the code for the same.
    3. The code is effectively frozen 
    git log gpcontrib/gpmapreduce
    shows that every commit since the initial Cloudberry code dump has been mechanical: rebranding, typo fixes, Python 3 migration, and PostgreSQL merge fallout. The last functional change dates back to the Greenplum era. 4. Users have better alternatives today The workloads gpmapreduce was designed for in 2008 are well covered by plain SQL plus UDFs (PL/pgSQL, PL/Python, PL/Perl, PL/Container), external tables / gpfdist, and FDWs. We are not aware of active users of the YAML job-spec interface, and this proposal is partly a call for such users to speak up. Implementation The change is confined to build glue and one self-contained directory. Proposed as a single PR with atomic commits: 1. Stop building and shipping 
    gpmapreduce
     — drop 
    --enable-mapreduce
    from 
    devops/build/automation/cloudberry/scripts/configure-cloudberry.sh
    (and update the feature comment at the top of that file), and remove the 
    mapreduce
    target plus the 
    install
    / 
    clean
    / 
    distclean
    branches from 
    gpcontrib/Makefile
    . 2. Delete 
    gpcontrib/gpmapreduce/
     — 44 files, ~13,500 lines. 3. Remove the configure surface — 
    --enable-mapreduce
    , 
    USE_MAPREDUCE
    , the `libyaml`/`yaml.h` checks that are specific to mapreduce, and the mapreduce-specific Perl requirement, in `configure.ac`; regenerate `configure`; drop 
    #undef USE_MAPREDUCE
    from 
    src/include/pg_config.h.in
    . 4. Remove the help file — 
    gpMgmt/doc/gpmapreduce_help
    and its entry in 
    gpMgmt/doc/Makefile
    . 5. Drop the RAT exclusion — remove 
    <exclude>gpcontrib/gpmapreduce/**</exclude>
    from 
    pom.xml
    . Rollout/Adoption Plan Version plan | Line | Behaviour | | ------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------- | | 2.x (REL_2_STABLE, 2.2.0-incubating and later 2.x releases) | No change. gpmapreduce continues to be built and shipped. Users who still depend on it stay on 2.x for the supported life of that line. | | 3.0 (main, 3.0.0-devel) | Removed. | | 3.x | Stays removed; not re-introduced. | This gives users an explicit, supported landing spot rather than an abrupt removal, while keeping the 3.x tree clean from the start of the line. Open questions for the community 1. Is anyone still running 
    gpmapreduce
    in production? If so, on which Cloudberry version, and what is the migration blocker? Are you willing to submit a PR? • Yes I am willing to submit a PR! apache/cloudberry
  • y
    Intermittent duplicate key violation on retry after "Broken pipe" error during cbcopy migration Hi all, I'm migrating data from a Greenplum source cluster to a Cloudberry destination cluster using cbcopy. I'm running into a recurring issue with one of our larger partitioned tables (~900M rows, range-partitioned into ~21 partitions) and would appreciate any guidance. Environment: • Source: Greenplum cluster • Destination: Cloudberry cluster Background / sequence of events: 1. Initial attempt — I ran a full schema data copy in one go with --copy-jobs=8. This caused CPU utilization on the source to spike to 100%, and during this run I also got a "Broken pipe" error on our largest table (~900M rows):    ERROR: could not write to COPY program: Broken pipe (copy.c:607) (seg0 <source_ip>:<port> pid=...) (SQLSTATE XX000) 1. Revised strategy — To reduce load, I changed approach: • First ran a schema-only migration (metadata only, no data) — this completed successfully with no issues. • Then planned to migrate all smaller tables together (--data-only --truncate, excluding the two large tables), followed by the two large tables one at a time, each with reduced --copy-jobs. 2. Large table attempt #1 — partition by partition — For the ~900M row table, I split the copy into individual partitions (--include-table per partition, --copy-jobs=1 or 2) to further reduce load. During this, the same partition failed with "Broken pipe" mid-copy. On retry of that same partition (after --truncate), I got:    ERROR: duplicate key value violates unique constraint "<table>_<partition>_<column>_idx1" (SQLSTATE 23505) 1. Large table attempt #2 — whole table at once — I also tried running the same large table as a single --include-table job (not split by partition). This produced the same pattern: intermittent "Broken pipe" followed by a duplicate key violation on retry, on a different partition than before. 2. Full schema re-migration — To rule out leftover/stale data, I dropped the entire destination schema and re-migrated it from scratch. The schema and smaller tables migrated fine. However, when copying the large table again, the same duplicate-key error recurred on yet another partition — despite this being a completely fresh destination with no prior data. Troubleshooting steps I've taken to rule out other causes: 1. Verified no duplicate values exist on the source table for the column backing the unique index (checked specifically against the single-column unique index, not just the composite one). 2. Confirmed source and destination partition boundaries are identical. 3. Confirmed indexes on source and destination match exactly (same indexes, same definitions). 4. Confirmed no other cbcopy process was running concurrently (ps -ef | grep cbcopy). 5. Manually truncated (and vacuumed) the destination table/partition before each retry. 6. Reduced --copy-jobs to 1 to minimize concurrency and CPU/network load. 7. Repeated the entire migration on a freshly dropped and recreated destination schema — the same duplicate-key error still occurred, just on a different partition. My working theory: Since cbcopy streams data directly segment-to-segment (via cbcopy_helper) rather than through a single atomic transaction, I suspect that when "Broken pipe" occurs, some segments may have already committed their portion of data before the connection dropped on another segment. If cbcopy's retry logic then re-attempts the copy without re-truncating (or without full transactional rollback across all segments), it would re-send rows that other segments already committed — producing a duplicate key violation against data it just wrote itself. Questions: 1. Is this a known issue with how cbcopy handles partial failures / retries at the segment level? 2. Is there a recommended way to make the truncate-and-copy operation atomic per table/partition, so a broken pipe mid-copy doesn't leave partially committed data behind? 3. Are there recommended settings (timeouts, retry behavior, --connection-mode, concurrency limits, etc.) to reduce the frequency of the underlying "Broken pipe" errors in the first place — particularly for large tables where source-side CPU/network load seems to be a contributing factor? 4. Is per-partition copying or whole-table copying generally more reliable for very large tables (~900M+ rows) in your experience? Happy to share full debug logs (with sensitive details redacted) if that would help diagnose this further. Any guidance would be greatly appreciated. Thanks!
    d
    t
    • 3
    • 4
  • d
    🎥 Apache Cloudberry Community Meeting Recording — Sept 18 The latest bi-weekly community meeting recording is available:

    youtube.com/watch?v=…▾

    Highlights include: • Cloudberry 2.2 release readiness and license fixes • Iceberg and Parquet support • Exploring agent-native analytics with MCP • Query optimization and selectivity improvements • Updates to PXF, Backup, Go Libs, Command Center, documentation, and ecosystem projects • Release planning and upcoming community activities Watch the recording and share your feedback with the community!
  • g
    #2043 [Proposal] Bundle `pgvector` as a built-in extension in Apache Cloudberry New discussion created by tuhaihe Proposers Dianjin Wang (@Dianjin Wang) Proposal Status Under Discussion Abstract Bundle pgvector as a first-class, built-in extension in Apache Cloudberry, following the existing 
    gpcontrib/
    pattern (the same way 
    orafce
    and 
    pg_hint_plan
    are vendored). Beyond just vendoring, the proposal includes the MPP integration work required to make vector search correct and well-documented. Motivation Vector search has become a table-stakes capability in the AI era, and PostgreSQL users expect pgvector to just work. Today Cloudberry users must install it separately from a maintained fork (cloudberry-contrib/pgvector). Bundling it gives us: • A single build/package/release pipeline and a pinned version per Cloudberry release. • Regression coverage in our own CI and installer. • A native, documented Cloudberry experience rather than an external add-on. There is already strong precedent: 
    orafce
    , 
    pg_hint_plan
    , and more cases are all third-party extensions vendored under 
    gpcontrib/
    . Implementation 1. Vendor into 
    gpcontrib/pgvector
    following the existing pattern: • Add 
    PGAC_ARG_BOOL(enable, pgvector, ...)
    in 
    <http://configure.ac|configure.ac>
    (mirroring 
    enable_orafce
    ). • Wire 
    pgvector
    into 
    gpcontrib/Makefile
    (
    recurse_targets
    , 
    install
    , 
    installcheck
    ). • Add 
    --enable-pgvector
    to the release 
    CONFIGFLAGS
    in 
    gpAux/Makefile
    and the devops configure script. 2. Do the MPP integration work that makes this more than a copy: • Introduce the pgvector codebase • Cherry-pick the commits that were made for Cloudberry in the 
    cloudberry-contrib/pgvector
    • Add the tests for pgvector in the CI workflows Rollout/Adoption Plan • vendor + build/package wiring, land as opt-in (
    --enable-pgvector
    ), add CI. Are you willing to submit a PR? • Yes I am willing to submit a PR! apache/cloudberry
  • s
    Subject: How does cbcopy handle a duplicate value conflict when the DISTRIBUTED BY column is also the PRIMARY KEY? Context: I'm migrating a production database from Greenplum 7 to Apache Cloudberry 2.0.1 using cbcopy. One of the tables (~880 million rows) has its DISTRIBUTED BY column also defined as the PRIMARY KEY. Question: If the source table contains duplicate values in this column — for example due to a constraint that was disabled or marked NOT VALID at some point on the source side — what is cbcopy's actual behavior when loading into a target table where the PRIMARY KEY constraint is active? Specifically: 1. Does cbcopy do any pre-check on this column before starting the load, or does it just attempt the insert and let the target's constraint layer catch it? 2. When a duplicate key violation happens mid-load, does the entire table/partition load abort and roll back, or does it skip the conflicting row(s) and continue with the rest? 3. Is there any flag or option to control this behavior — for example, skipping and logging conflicting rows versus a hard failure? I'd like a definitive answer before running this on a table this large in production, rather than relying on trial-and-error. Any insight from someone who has hit this scenario during a large-scale migration would help a lot. Environment: • Source: Greenplum 7.x • Target: Apache Cloudberry 2.0.1 • Table size: ~880 million rows • Column: DISTRIBUTED BY + PRIMARY KEY (same column)
  • g
    #2050 How does cbcopy handle a duplicate value conflict when the DISTRIBUTED BY column is also the PRIMARY KEY? New discussion created by asheraz048i2c Context: I'm migrating a production database from Greenplum 7 to Apache Cloudberry 2.0.1 using cbcopy. One of the tables (~880 million rows) has its DISTRIBUTED BY column also defined as the PRIMARY KEY. Question: If the source table contains duplicate values in this column — for example due to a constraint that was disabled or marked NOT VALID at some point on the source side — what is cbcopy's actual behavior when loading into a target table where the PRIMARY KEY constraint is active? Specifically: Does cbcopy do any pre-check on this column before starting the load, or does it just attempt the insert and let the target's constraint layer catch it? When a duplicate key violation happens mid-load, does the entire table/partition load abort and roll back, or does it skip the conflicting row(s) and continue with the rest? Is there any flag or option to control this behavior — for example, skipping and logging conflicting rows versus a hard failure? I'd like a definitive answer before running this on a table this large in production, rather than relying on trial-and-error. Any insight from someone who has hit this scenario during a large-scale migration would help a lot. Environment: Source: Greenplum 7.x Target: Apache Cloudberry 2.0.1 Table size: ~880 million rows Column: DISTRIBUTED BY + PRIMARY KEY (same column) apache/cloudberry
  • g
    #2065 [Proposal] Apache Cloudberry as PostgreSQL 19 Extensions New discussion created by igor-suhorukov Proposers Igor Suhorukov (@igor-suhorukov) Proposal Status Under Discussion Abstract I would like to discuss running Apache Cloudberry on PostgreSQL 19 as a set of extensions on top of 22 dormant core hooks, instead of a forked kernel. A working prototype exists; it is my personal experiment, not an ASF release. Without the modules, the patched server is measurably vanilla PostgreSQL 19: the same PostgreSQL tests pass, the ABI is unchanged, catalogs are byte-identical, and the hooks add less than 0.2% to executed instructions. With them, it is a Greenplum-style MPP engine with plan slices, Motions, two-phase commit and distributed snapshots, running unmodified ORCA and stock PostGIS and pgvector. ORCA plans all 121 TPC-H and TPC-DS queries at SF1 with answers matching DuckDB's, and 691 of Cloudberry's 1,180 test files pass so far. Claude Code wrote the code over 12 days and 629 commits; I set the goals and signed off every change to the PostgreSQL core. Motivation Greenplum and Cloudberry have always followed PostgreSQL a few releases behind: Greenplum 6 shipped on PostgreSQL 9.4, Greenplum 7 on 12, Cloudberry 2.x on 14, and 
    main
    reached 16.9 in May 2026. The reason is structural. Cloudberry edits 882 PostgreSQL backend and header files, so every new PostgreSQL major is a large merge. #1095 estimated the 14 → 16 upgrade at 4–5 months, three of them for fixing regression tests, and noted that PostgreSQL 14, the base of Cloudberry 2.x, reaches end of life in 2026. Meanwhile PostgreSQL 17 and 18 have shipped, and 19 is close to release. In #1877, the project named close compatibility with modern PostgreSQL as one of its goals. The prototype tests whether that goal can become cheap to sustain. If Cloudberry runs as extensions, the next PostgreSQL major means rebasing a small patch series instead of merging a fork. Users get a current PostgreSQL, including PostgreSQL 19 features such as eager aggregation and parallel autovacuum. They also get stock extensions: upstream PostGIS 3.7 instead of a PostGIS fork, and pgvector 0.8.6 built unpatched, which is relevant to #2043. The prototype compiles Cloudberry's own sources in place, so it builds on the community's work rather than replacing it. Implementation Details: Greenplum Without the Fork. Code: core series, port, build instructions. The ground rule: a server with the core patches but without Cloudberry loaded must be indistinguishable from vanilla PostgreSQL 19, and this is measured. • Core series: 22 commits, 47 files, +1,578/−41 lines. Existing extension points come first: hooks, table AMs, CustomScan, custom WAL resource managers, background workers, security labels and FDWs. Where they are not enough, a new hook is a NULL-by-default function pointer, a new exported function, a registration list or an off-by-default flag. No existing signature changes, no struct gains a field, and no catalog, WAL, page or protocol format changes. • Vanilla checks compare Docker images built from the same PostgreSQL commit with and without the series: • the same 371 
    meson test
    results; • `abidiff`: 32 symbols added, 0 changed; • byte-identical catalogs, settings, keywords and stored parse trees; • identical query IDs; • interchangeable data directories and streaming replication; • cachegrind: −0.005% to +0.155% executed instructions; • a script that checks every statement added inside an existing function sits behind a hook, registry or flag, with one reviewed exception. • Cloudberry's tree is untouched. The whole port is one new directory, 
    pg19/
    . Cloudberry's files compile where they lie, so merges from Cloudberry keep applying, and a file that must change becomes a copy that says what changed. ORCA's 920 core files compile unmodified behind 
    planner_hook
    , and PAX compiles in place with 65 of its 91 files unchanged. • Modules: • `gp_core`: dispatcher, Motions, interconnect, 2PC, distributed snapshots, GDD, FTS; • `gp_orca`; • `gp_sql`: Cloudberry syntax rewritten into PostgreSQL 19's grammar; • storage: 
    gp_ao
    , `pax`; • external data: 
    gp_exttable
    , 
    pxf_fdw
    , 
    gpcloud
    , `datalake_fdw`; • 
    gp_resource
    , 
    gp_security
    , 
    gp_matview
    , 
    gp_task
    , `diskquota`; • 
    gpfts
    and gpMgmt's tools from 
    gpinitsystem
    to 
    gpexpand
    . • What changes for users: • migration is logical (dump and restore); • settings are renamed 
    gp.*
    , although 
    gpconfig
    accepts the old names; • DDL command tags become standard; • the 20 shared catalogs are replaced by security labels, extension tables and a cluster file; • TDE gives way to volume encryption. Rollout/Adoption Plan Nothing here affects Cloudberry 2.x or the 3.0 plans: the prototype lives in my fork and changes no lines in Cloudberry's files. I would like to ask the community for four things: 1. Feedback on the direction. Is running Cloudberry as PostgreSQL extensions worth pursuing as a long-term path after 3.0? 2. A home for 
    pg19/
    .     It is one directory that tracks Cloudberry's sources without editing them. If the direction looks right, would the project consider hosting it, for example as an experimental branch or a separate repository? 3. Help with the remaining tests. The goal is to pass Cloudberry's original test suites with minimal changes to the tests. Every skipped test is listed with the statement that stops it, so tests are easy to pick up; 
    isolation2
    and 
    greenplum_schedule
    are next. 4. A decision on Route B. Should the port bring over Cloudberry's MPP variant of the PostgreSQL planner (30–45k lines), or keep shrinking ORCA's declines? On the regression suite, the port's own fallback reasons already fell from 563 to 52. A proposed 
    gp.mpp_planner
    switch would allow side-by-side comparison, and I would value the view of the planner experts here. For users, migration would be logical, similar to the cluster-to-cluster copy planned for 2.0 → 3.0 in #1095. Separately, four of the custom hooks (table AM registry, smgr file events, parser hook and OID hook) are general enough to propose to PostgreSQL upstream. Support from Cloudberry developers as their real-world user would strengthen that case. Are you willing to submit a PR? • Yes I am willing to submit a PR! apache/cloudberry
  • g
    #2108 [Ideas] pluggable cumulative statistics in Cloudberry New discussion created by Alena0704 Description Proposal: pluggable cumulative statistics in Cloudberry The idea and intended result Pluggable cumulative statistics let extensions define their own statistics and store them through the same infrastructure as built-in statistics. An extension registers a statistics kind describing its data, and the common system provides shared-memory storage, reads, resets, and persistence across clean restarts. We want new statistics and their SQL interfaces to evolve in extensions. Adding an extension-owned metric should not require adding a built-in catalog function or system-view column and bumping the core catalog version for that change. Extensions should also be able to reuse the common statistics store instead of implementing storage, synchronization, resets, and persistence independently. Pluggable statistics provide the storage mechanism. The extension author decides what to count, where to update the counters, and how to expose them through SQL. New measurement points may still require core hooks. How extensions and users would use it An extension developer defines counters, registers their kind at server startup, and connects measurement code or reporting hooks to updates of those counters. The extension supplies SQL functions and views for reading the values, plus any collection controls and reset functions it needs. An operator loads the module through 
    shared_preload_libraries
    , restarts the server, and installs the extension's SQL objects in the relevant databases. Monitoring then queries the extension's views and compares observations over time, accounting for resets. In Cloudberry, the extension also defines how local observations are exposed and aggregated across the coordinator and segments. This gives each consumer control over its metrics and SQL interface while keeping storage and lifecycle operations in the common cumulative-statistics system. The VACUUM example below follows this path from measurement to SQL. What Cloudberry has today and why it is insufficient Before the proposed port, Cloudberry already has a cumulative-statistics infrastructure for built-in kinds, including Cloudberry-specific resource-queue statistics. What is missing is the supported registration API that lets extensions define their own kinds and participate in that infrastructure. There are two existing approaches to adding statistics, each with a limitation: | Approach | Limitation | | ------------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------ | | Add a built-in statistics kind or extend existing core statistics and SQL interfaces | Metric-specific changes become core changes; adding catalog functions or system-view columns also requires catalog maintenance | | Keep statistics in an extension-owned store | The extension must maintain its own storage, synchronization, reset, and persistence behavior | For example, 
    pg_stat_statements
    maintains its own shared hash table with a limit set by 
    pg_stat_statements.max
    and removes entries when it fills up. This illustrates the separate storage machinery an extension maintains today. pg_stat_statements implementation The common cumulative-statistics system also uses a hash table, backed by dynamic shared memory. It allocates memory as entries are added, without a configured entry-count limit like 
    pg_stat_statements.max
    . Making this store available to custom kinds would let extensions reuse it. Statistics memory allocation Proposed implementation and how it works This mechanism has already been added to PostgreSQL and is documented in PostgreSQL 19. PR #2102 proposes bringing it to Cloudberry by cherry-picking the upstream commits and adapting them, including preservation of Cloudberry-specific resource-queue statistics. The implementation extends the existing cumulative-statistics infrastructure with registration and access APIs. Its operation and division of responsibilities are described first, followed by the commits that introduce them. Registration and storage An extension describes a statistics kind with 
    PgStat_KindInfo
    and registers it through 
    pgstat_register_kind()
    while loading through 
    shared_preload_libraries
    . Two object models are supported: | Model | Example use | | ---------------------------- | -------------------------------------------------- | | Fixed-numbered statistics | A fixed set of counters for a subsystem | | Variable-numbered statistics | Separate entries for tables, indexes, or databases | The kind descriptor defines the data layout, storage properties, and applicable callbacks. Extensions can use pending local updates with callbacks to flush them into shared memory. The 
    write_to_file
    property controls whether statistics participate in persistence. The PR also adds an optional 
    reset_data_cb
    for variable-numbered kinds. It lets a reset clear counters while preserving identifying metadata in the entry payload. Without the callback, the existing zero-fill behavior remains. PR #2102 Limitations Dynamic allocation does not guarantee that a statistics update will succeed when memory is exhausted. Variable-numbered kinds share the cumulative-statistics memory infrastructure. Their memory use grows with the number of tracked objects and the size of each entry, with additional overhead for the shared hash table, backend-local references, pending updates, and snapshots. There is no per-kind entry-count limit or automatic eviction policy comparable to 
    pg_stat_statements.max
    . When a new entry is needed, the current implementation inserts it into the shared hash table and allocates its payload with 
    dsa_allocate0()
    . This allocation does not use `DSA_ALLOC_NO_OOM`: if memory cannot be allocated, the allocator raises 
    ERROR
    . The infrastructure therefore does not guarantee that it will simply skip a statistics sample under memory pressure. The error can propagate to the operation collecting the statistics, including a VACUUM reporting through an extension hook. See the local implementation in <../src/backend/utils/activity/pgstat_shmem.c|pgstat_shmem.c>, <../src/include/utils/dsa.h|dsa.h>, and <../src/backend/utils/mmgr/dsa.c|dsa.c>. This makes memory budgeting and cleanup part of extension design. Extensions should keep payloads compact, avoid unnecessarily high-cardinality keys, and connect object deletion to entry cleanup. A per-kind quota, eviction policy, or a safe way to drop a sample on allocation failure would require additional design; the proposed API does not provide these guarantees. Memory-pressure tests should establish both the error seen by the caller and whether subsequent collection remains usable. Other limitations also affect consumers: • Custom kind IDs are limited to 24–32 and must be unique across loaded extensions. Registration happens during preload, so enabling a new module requires a restart. • Shared-entry locking, pending-update flushing, and snapshot creation have a cost. The API does not guarantee negligible overhead under frequent updates or concurrent access. • Re… apache/cloudberry
  • g
    #2118 [Proposal] New discussion created by Alena0704 Proposal Status Under Discussion Abstract I propose introducing VACUUM metric collection in Cloudberry to evaluate the work and resource cost of VACUUM over time and measure the effect of autovacuum tuning. The proposal adds measurements and statistics to core and exposes them partly through built-in functions and views and partly through 
    ext_vacuum_statistics
    . Basic timing and work counters remain available independently of the extension. The extension accumulates additional buffer, I/O, and WAL measurements, exposes them alongside native counters, and provides a way to add further metrics and collection controls. Cumulative statistics for tables, indexes, and databases would support analysis of maintenance trends, including differences between segments. SQL access allows users to analyze the statistics directly or collect historical samples with their preferred monitoring tool. Integration with 
    pg_profile
    is proposed as an optional follow-up. A partial implementation for 
    REL_2_STABLE
    exposes its statistics through a separate 
    vacuum_stats
    extension without changing the system catalog version. Motivation Understand the results and cost of maintenance VACUUM must keep up with dead tuples, freezing, and visibility-map maintenance while sharing resources with application queries. The appropriate autovacuum settings depend on each table's workload and its indexes. Administrators need measurements that show both the useful work performed and its cost. This is the motivation described in the initial PostgreSQL proposal and the follow-up explanation of per-relation counters. Existing invocation counts and timestamps show whether maintenance ran. Progress views describe operations currently running. Accumulated work and resource measurements would provide the history needed to answer questions such as: • Which tables and indexes account for most VACUUM time, buffer activity, and WAL generation? • Do repeated runs remove tuples, or keep encountering versions that cannot yet be removed? • Does a run with few removals still perform useful freezing or visibility-map work? • How much elapsed time comes from cost-based delays? • Does AO/AOCO compaction remove hidden rows, and what work remains afterwards? • Are maintenance costs or unfinished cleanup concentrated on particular segments? These measurements would help select relations for investigation and assess changes to their autovacuum settings. Activity views, logs, and physical inspection would supply the additional evidence needed to establish causes. Analyze changes over time Regular samples of cumulative counters would allow users to compare increments and growth rates over successive intervals. This would show how maintenance cost changes over hours, days, and weeks, how that change relates to application workload, and whether tuning has a sustained effect. Comparing periods before and after a settings change or REINDEX is one application of this history. As an optional follow-up, I propose adding the new metrics to pg_profile, which provides a historical sample repository and interval reports. Users could also query the SQL views directly, save their own samples, or use other monitoring tools. The core and extension statistics do not depend on 
    pg_profile
    . Make statistics extensible The extension provides a place to develop additional metrics and their SQL interfaces without requiring a core catalog change for each new field. Measurements already supplied by core can be accumulated and exposed through an extension update. New measurement points may still require core changes. The extension also provides a place to develop flexible collection policies, such as collecting additional statistics only on selected segments or in selected databases, or enabling collection for selected extensions. These filters are proposed future work. They would let operators choose the scope of additional collection independently of the basic native statistics. Implementation Series for 
    main
     The implementation separates instrumentation, native cumulative counters, and additional statistics owned by the extension. | PR | Contribution | | ------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------- | | [#2102](https://github.com/apache/cloudberry/pull/2102) | Pluggable cumulative statistics, allowing extensions to reuse the common storage, reset, and persistence infrastructure | | [#2103](https://github.com/apache/cloudberry/pull/2103) | Optional VACUUM buffer, I/O, and WAL instrumentation and resource reporting under VACUUM (VERBOSE) | | [#2104](https://github.com/apache/cloudberry/pull/2104) | Native maintenance timing, cost-delay, visibility-map, failsafe, and interruption statistics; pg_class.relallfrozen | | [#2105](https://github.com/apache/cloudberry/pull/2105) | Native VACUUM work statistics for heap, indexes, and AO | | [#2106](https://github.com/apache/cloudberry/pull/2106) | The reporting hook and ext_vacuum_statistics, including local and cluster resource views, summaries, and distributed resets | | [#2107](https://github.com/apache/cloudberry/pull/2107) | Detailed heap reporting and AO/AOCO reporting, including separate views of resource use by AO phase | Core accumulates basic timing and work counters and exposes them through built-in SQL interfaces. The extension reads the native work counters alongside its own resource statistics, without accumulating those native counters again. The consumer hook is introduced with the extension in PR #2106; the instrumentation in PR #2103 can report measurements through verbose logging independently. PR #2104 also adds 
    pg_class.relallfrozen
    , an estimate of the number of heap pages marked all-frozen in the visibility map. Compared with 
    relpages
    and 
    relallvisible
    , it helps assess freezing coverage and tune freezing behavior. VACUUM can skip all-frozen pages even during an anti-wraparound scan while their marks remain set. This catalog value records the state observed at the last refresh by operations such as VACUUM or ANALYZE; it is separate from cumulative freezing-work counters. Extension storage and additional metrics On 
    main
    , 
    ext_vacuum_statistics
    registers relation and database statistics kinds with the pluggable 
    pgstat
    infrastructure. The common system supplies dynamically allocated shared storage, reads, resets, and persistence across compatible clean restarts. This avoids maintaining a separate fixed-capacity hash table and its entry-eviction policy. Memory use still grows with the number and size of tracked entries. The <pluggable-statistics.md|pluggable-statistics discussion> describes the storage API and its constraints. The extension accumulates buffer reads, hits, dirtying and writes, measured I/O time, and WAL records, full-page images, and bytes. Table and index resource accounting is kept separate. AO reporting includes dedicated views for the PRE_CLEANUP, COMPACTION, and POST_CLEANUP phases, alongside native compaction work counters. Auxiliary heap relations retain statistics under their own OIDs. Collection requires loading 
    ext_vacuum_statistics
    through `shared_preload_lib… apache/cloudberry
  • g
    #2128 [Ideas] Add ORCA intra-segment parallel scan support for partitioned AO tables New discussion created by Alena0704 Description
    Preliminary measurements: These are noisy, uncontrolled measurements collected on my laptop with background applications running, not a clean benchmark. I plan to repeat the measurements on a virtual machine with a cold cache. Treat the timings and speedup ratios below as provisional.
    For 
    SELECT count(*) FROM ao13
    over a row-oriented append-optimized table with 13 daily partitions and 6 million rows, ORCA uses a serial 
    Dynamic Seq Scan
    within each segment even when 
    enable_parallel=on
    and 
    max_parallel_workers_per_gather=4
    . The PostgreSQL planner produces 
    Parallel Append
    over 
    Parallel Seq Scan
    . Disabling 
    optimizer_enable_dynamictablescan
    causes PostgreSQL fallback for this query. The parallel PostgreSQL plan reduces median execution time from 751.380 to 659.712 ms on unmodified upstream main (12.2% less time; 1.14× speedup) and from 713.039 to 649.303 ms on unmodified upstream REL_2_STABLE (8.9% less time; 1.10× speedup), with 
    gp_resqueue_priority=off
    . Parallel Append lets multiple execution participants within each segment share the partition-scan work, rather than having one participant scan all partitions sequentially. The measured parallel plans have 12 sending participants across three physical segments, versus three for ORCA. This distributes scan and partial-aggregation work across more CPU cores; scheduling overhead, work granularity and host load limit the resulting speedup. The fallback uses the same PostgreSQL parallel execution mechanism. The limitation reproduces on both unmodified upstream branches. This appears to be a missing capability in ORCA's intra-segment parallel scan support, rather than a cost-model choice between serial and parallel dynamic scan alternatives. Parallel execution improves this query's elapsed time in these measurements, but does not close the performance gap to GP6 on the unmodified builds; the cross-system comparison is included below. Every timing and plan in this report was collected with 
    gp_resqueue_priority=off
    on the coordinator and all three primary segments.     Each mode has one warm-up followed by five measured runs. Tables report execution times in milliseconds; full plans are from measured round 5. Environment and procedure • Test date: October 14, 2026; macOS arm64 host, 14 logical CPUs, 24 GiB RAM. • Each cluster has three primary segments on the same host and database 
    codex_ao13_verify_20261014
    . • Same table definition, data generation, distribution on 
    id
    , 13 daily partitions and 6 million rows across builds. • 
    gp_resource_manager=queue
    , 
    gp_resqueue_priority=off
    , 
    statement_mem=128000
    kB and 16 MiB shared buffers. • Cloudberry: 
    max_parallel_workers_per_gather=4
    , 
    optimizer_disable_dynamic_table_scan=off
    in every mode. • One persistent psql session per build; one warm-up EXPLAIN per mode, then five measured rounds, with mode order shuffled using seed 1306. Round 0 is excluded from statistics. • The build benchmarks ran sequentially. No other benchmarks or builds were launched by this investigation during measurement. Other host applications remained running; process-load samples were recorded every two seconds. This is not an isolated-host benchmark. • No OS cache flushing was performed. An explicit count was checked once per mode during warm-up: all benchmark count checks returned 6000000. 
    gp_resqueue_priority
    is a startup setting. It was disabled through postgres startup options on every node, with cluster restart where necessary; it was not changed using a session SET. The setting was verified on every node (direct utility connections for the clean Cloudberry clusters; coordinator SHOW plus distributed pg_settings queries for GP6 and the PR build), and through SHOW in each benchmark session. The test clusters remain running with this startup override. Cluster startup option and setting verification
    Copy code
    # Add this option to the existing postgres startup arguments on every node:
-c gp_resqueue_priority=off
    SHOW gp_resqueue_priority; -- must report off The exact restart and sequential measurement procedure is in <parallel-partition-verification/analysis/run_priority_off_comparison.py|run_priority_off_comparison.py>. Preserve the appropriate dispatch/execute role arguments when restarting nodes. Reproduction SQL Create and populate this table in each test database where 
    ao13
    does not already exist: Create, populate and analyze ao13 CREATE TABLE ao13 ( id bigint, d date, col1 int, col2 int, pad text ) WITH (appendonly=true, orientation=row) DISTRIBUTED BY (id) PARTITION BY RANGE (d) ( START ('2020-01-01'::date) END ('2020-01-14'::date) EVERY ('1 day'::interval) ); INSERT INTO ao13 SELECT g, '2020-01-01'::date + (g % 13), g % 97, g % 89, repeat('x', 50) FROM generate_series(1, 6000000) g; ANALYZE ao13; The SQL for each optimizer mode accompanies its plan below. In Cloudberry, disabling 
    optimizer_enable_dynamictablescan
    prevents a specific ORCA transformation; fallback is verified for this query, not guaranteed for arbitrary queries. The distinct 
    optimizer_disable_dynamic_table_scan
    remains 
    off
    . Setting it to 
    on
    enables ORCA's existing dynamic-scan-to-Append translation, which does not itself provide the parallel alternative described here. Transaction-scoped fallback reproduction BEGIN; SET LOCAL optimizer = on; SET LOCAL optimizer_enable_dynamictablescan = off; SET LOCAL optimizer_disable_dynamic_table_scan = off; SET LOCAL enable_parallel = on; SET LOCAL max_parallel_workers_per_gather = 4; EXPLAIN (ANALYZE, COSTS OFF, TIMING OFF) SELECT count(*) FROM ao13; COMMIT; Cloudberry unmodified upstream main Source commit 
    3a586f12c2c2011989734be383ddea69aac24abb
    , verified against upstream main on October 14, 2026. All 19,632 tracked files and symbolic links matched the commit; no local feature patches or PR #2094 code were applied. Cloudberry 3.0.0-devel / PostgreSQL 16.9, Apple Clang 21, 
    --enable-debug --enable-cassert --enable-orca
    , 
    CFLAGS='-O0 -g'
    , 
    CXXFLAGS='-O0 -g'
    . ORCA, parallel on Reproduction SQL SET optimizer = on; SET optimizer_enable_dynamictablescan = on; SET optimizer_disable_dynamic_table_scan = off; SET enable_parallel = on; SET max_parallel_workers_per_gather = 4; EXPLAIN (ANALYZE, COSTS OFF, TIMING OFF) SELECT count(*) FROM ao13; Query execution plan — measured round 5
    Copy code
    Finalize Aggregate (actual rows=1 loops=1)
   ->  Gather Motion 3:1  (slice1; segments: 3) (actual rows=3 loops=1)
         ->  Partial Aggregate (actual rows=1 loops=1)
               ->  Dynamic Seq Scan on ao13 (actual rows=2001127 loops=1)
                     Number of partitions to scan: 13 (out of 13)
                     Partitions scanned:  Avg 13.0 x 3 workers.  Max 13 parts (seg0).
 Planning Time: 10.136 ms
   (slice0)    Executor memory: 114K bytes.
   (slice1)    Executor memory: 293K bytes avg x 3x(0) workers, 293K bytes max (seg0).
 Memory used:  128000kB
 Optimizer: GPORCA
 Execution Time: 751.380 ms
    PostgreSQL, parallel off Reproduction SQL SET optimizer = off; SET optimizer_enable_dynamictablescan = on; SET optimizer_disable_dynamic_table_scan = off; SET enable_parallel = off; SET max_parallel_workers_per_gather = 4; EXPLAIN (ANALYZE, COSTS OFF, TIMING OFF) SELECT count(*) FROM ao13; Query execution plan — measured round 5 ``` Finalize Aggregate (actual rows=1 loops=1) -> Gather Motion 3:1 (slice1; segments: 3) (actual rows=3 loops=1) -> Partial Aggregate (actual rows=1 loops=1) -> Append (actual rows=2001127 loops=1) -> Seq Scan on ao13_1_prt_1 ao13_1 (actual rows=154105 loops=1) -> Seq Scan on ao13_1_prt_2 ao13_2 (actual rows=154176 loops=1) -> Seq Scan on ao13_1_prt_3 ao13_3 (actual rows=154082 loops=1) -> Seq Scan o… apache/cloudberry
  • d
    🎊*️ Cloudberry 3.0 release planning is underway.* This will be the first Cloudberry release following the PostgreSQL 16 kernel upgrade. The current discussion covers: • Completing the remaining PostgreSQL 16 work • Porting extensions such as 
    yezzey
    • Coordinating compatibility with cloudberry-backup, cloudberry-pxf, cloudberry-go-libs, and other ecosystem projects • Defining upgrade paths from Cloudberry 2.x and older Greenplum versions • Creating the 
    REL_3_STABLE
    branch • Improving CI and build support, including possible Meson and Ubuntu 26.04 support • Evaluating Anser and 
    datalake_fdw
    We would especially appreciate feedback from users and maintainers on upgrade requirements, compatibility issues, testing priorities, and potential release blockers. Please join the discussion here: lists.apache.org/thread/…
  • g
    Comment on #2128 [Ideas] Add ORCA intra-segment parallel scan support for partitioned AO tables Discussion answered by yjhjstz Thanks for the very thorough report, @Alena0704. Your code reading matches what is on main today, and the conclusion is right: this is a missing capability in ORCA, not a cost-model decision. Current state • PR #1398 deliberately scoped intra-segment parallelism to plain CLogicalGet. CLogicalDynamicGet::PxfsCandidates() still only offers ExfDynamicGet2DynamicTableScan and ExfExpandDynamicGetWithForeignPartitions, so ORCA never generates a parallel alternative for a partitioned table. ExtractParallelWorkersFromDXL() returning 1 for EdxlopPhysicalDynamicTableScan is consistent with that, not a separate bug. • The fallback you observed with optimizer_enable_dynamictablescan=off is expected. With that xform disabled, CLogicalDynamicGet has no physical implementation, ORCA fails to find a plan, and we fall back to the PostgreSQL planner. It happens to produce Parallel Append here, but it is not a supported way to ask ORCA for a parallel plan. • This gap is part of the roadmap in discussion #1316, and dynamic scans are the natural next milestone after #1398. I already have a draft PR for this and will open it shortly. It follows the second direction you described: reuse the dynamic-scan-to-Append translation and the existing Parallel Append executor, instead of making DynamicSeqScan parallel-aware. The main reasons: 1. CTranslatorExprToDXL::PdxlnDynamicScanToAppend() already emits the Append with the partition selector ids of the scan, so join-driven dynamic partition pruning is preserved. 2. Parallel Append over Parallel Seq Scan children gives both whole-partition scheduling and intra-partition parallelism. The AO parallel scan already splits work by segment file, so a single surviving partition still parallelizes, which a shared partition counter inside DynamicSeqScan would not do. 3. No new DSM walker is needed in execParallel.c; Append and SeqScan are already handled. apache/cloudberry
  • d
    👋 Apache Cloudberry 2.2.0 — where things stand We’ve cut two release candidates so far, and neither has passed the vote. Both were blocked on LICENSE and NOTICE compliance issues found during review. Being blunt about it: this is normal, and it’s the process working. Apache releases make a promise about the licensing and provenance of every file we ship, and our mentors and the IPMC check that carefully. Finding these during a vote is much better than finding them after. We’re working through the remaining items — RC3 is close. If you want to help rather than wait: • Read the 2.2.0 preview so you know what’s coming → cloudberry.apache.org/blog/apache-cloudberry-2.2.0-preview • When RC3 is out, build it, test it, and reply on the vote thread. Non-binding +1s and -1s from users are genuinely useful — they tell us whether this thing works outside CI. Thanks for your patience. 2.2.0 is a big one: PostgreSQL kernel 14.4 → 14.10 with 41 CVE fixes, six new extensions, PXF observability, and the first release of Cloudberry Go Libs.
  • g
    #2146 [Proposal] Additional labels and a label suggestion tool New discussion created by Alena0704 Description I suggest adding a few labels to make issues and PRs easier to browse and filter. PR availability for issues It would be useful to see directly in the issue list whether an issue already has an associated PR. A 
    has-pr
    label could make this visible without opening each issue and allow filtering by PR availability. Ideally, this label would be applied automatically. We should also agree on how to handle it when a PR is closed without merging while the issue remains open. Topic labels for issues and PRs At a minimum, I propose adding: • 
    postgres optimizer
    — PostgreSQL query optimizer topics; especially it is strange that we don't have it since Orca label is available. • 
    vacuum
    — VACUUM and autovacuum; • 
    statistics
    — statistics collection and usage, including ANALYZE. Other potentially useful labels: • 
    wal
    — write-ahead logging; • 
    backup
    — backup-related topics. These labels would help contributors find related issues and PRs by subsystem and identify changes relevant to their interests or expertise. Label suggestion tool It could also be useful to introduce a tool that suggests relevant labels based on the title and description of an issue or PR. The tool could recommend existing labels when an issue or PR is opened or updated, leaving the final choice to the author or maintainers. This could make labeling more consistent and reduce manual work. Discussion • Any suggestions for naming or other missing categories? • Would a label suggestion tool be helpful, and are there any existing tools we could use? Use case/motivation No response Related issues The topic of additional labels and automated bug classification has already come up in Discussion #1952 — Continuous SQLancer fuzzing, which proposes a 
    sqlancer
    label and classification using 
    type: Orca
    or 
    planner
    . Are you willing to submit a PR? • Yes I am willing to submit a PR! apache/cloudberry
  • g
    Comment on #2118 [Proposal] VACUUM statistics for performance analysis Discussion answered by jianlirong I strongly support this proposal. Improving VACUUM statistics is a valuable step toward making Apache Cloudberry more observable and easier to operate in production. It aligns well with the broader goal of continuously improving the observability of the system, enabling DBAs to better understand maintenance costs, diagnose bottlenecks, and evaluate the effectiveness of autovacuum tuning. As we move forward, I think a few aspects deserve particular attention: • Statistics accuracy and semantics: Clearly define what each metric represents, especially the distinction between cumulative work and current state. • MPP-level visibility: Preserve per-segment statistics alongside cluster-wide aggregates, so that local bottlenecks are not hidden by global averages. • Production overhead and reliability: Carefully evaluate the overhead of collecting statistics and ensure correct behavior across resets, restarts, and segment failures. • Historical analysis: Consider integration with tools such as pg_profile to make it easier to track trends and compare maintenance performance over time. Overall, this is a meaningful improvement, and I look forward to seeing it evolve into a practical part of Cloudberry's operational observability toolkit. Thanks for the great work! Lirong apache/cloudberry
  • g
    #2157 [DISCUSS] Where should topic branches live: apache/cloudberry or personal forks? New discussion created by my-ship-it
    This is a pointer to a thread on dev@cloudberry.apache.org. Please reply on the mailing list so the discussion stays in one place:
    https://lists.apache.org/thread/89jhpr74ck9nq486h8c9vdm4sth6y2cp
    Not subscribed? Send an empty email to dev-subscribe@cloudberry.apache.org, or reply from the archive page above after signing in with your Apache ID.
    Summary apache/cloudberry has 21 branches today. Besides 
    main
    and the release branches, most were created for a single PR (
    fix_issue_1915
    , 
    cp_CVE_2025_12817
    , 
    bp_old_main
    , 
    reshke-patch-1
    , ...), and some names don't say what the branch is for at all (
    ijjdiew
    , 
    PG_
    ). The proposal: 1. The main repo keeps 
    main
    , 
    REL_*_STABLE
    , archive branches, and shared 
    feature/*
    branches for multi-person work agreed on dev@. 2. Everything else (bug fixes, single-author features, cherry-picks, backports) goes in personal forks. 3. One-time cleanup: owners have two weeks to claim branches they still need; the rest get deleted. PR history stays in 
    refs/pull/*
    . If you own one of the current branches and still need it, please say so on the thread. apache/cloudberry
  • d
    <!everyone> The recording of our October 16 bi-weekly community meeting is available:

    youtube.com/watch?v=HzFhJEQPixU▾

    Highlights include: • Repository activity and recent PRs • Vacuum and pluggable statistics • Optimizer and query-progress improvements • Iceberg/data-lake and MCP work • PXF updates and performance improvements • SQLancer testing and release-readiness work • Upcoming community and project milestones Thanks to everyone who contributed to the discussion. Please watch the recording and share your feedback!