Hi All, We are evaluating Apache Cloudberry (Incubating) for a production deployment and are looking for commercial support options. If you provide Cloudberry support services or can recommend a support partner, please DM me.

📢 New Migration Resource Available If you're planning a migration from Greenplum 6.x to Apache Cloudberry, check out this new guide contributed by community member vovik0134: 📖 cloudberry.apache.org/docs/sys-admin/migration-and-upgrade/backward-incompatible-changes-from-greenplum-6 The document summarizes compatibility changes across: • PostgreSQL kernel upgrades • SQL syntax and function behavior • Partitioning mechanisms • System catalogs and metadata • Configuration and resource management Every section includes: 🔍 Detection queries 🛠️ Recommended remediation steps This guide reflects many years of PostgreSQL evolution between Greenplum 6 (PostgreSQL 9.4-based) and modern Apache Cloudberry releases, making it especially useful for organizations planning a large-version upgrade. Huge thanks to vovik0134 for contributing this valuable resource to the Apache Cloudberry community! 🎉

📺 Apache Cloudberry Bi-weekly Community Meeting Recording is Available! The Apache Cloudberry ecosystem continues to grow. Recent highlights include: • 🚀 PostgreSQL 16 stabilization • 🔍 New extensions for interconnect observability and relation size monitoring • 🛡️ A new extension to prevent costly partition full scans • 🔐 Continuous CVE security updates • 📖 New Greenplum → Cloudberry migration guide • ⚙️ Progress on PXF, Cloudberry Backup, Release 2.2, and more Watch the recording to learn what's happening across the community! 🎥 📓 docs.google.com/document/d/14NLYVvApvij…/edit?tab=t.wvg9yii82vhl

Hello everyone, As I mentioned in my introduction post, I've created an kubernetes operator for Apache Cloudberry. The first public version was released today. Main features in this release: • Automatic coordinator / standby takeover • Automatic segment / mirroring takeover • Segment scale-up • Short-lived SSH certificates • And more I'd love to get your feedback and help make the operator more robust, so please test it out, and don't hesitate to reach out if you run into any issues. github.com/keldonio/keldon-operator

#1838 Keldon — Kubernetes Operator for Apache Cloudberry (looking for feedback) New discussion created by elhimad Hi everyone, I've been building Keldon, an open-source Kubernetes operator for Apache Cloudberry, and just shipped the first public release (v0.11.1). What it does: • Fully automated cluster bootstrap via

gpinitsystem

— coordinator, standby, segments, and mirrors, all provisioned declaratively from a single YAML • Horizontal scale-up: patch

spec.segments.count

, the operator provisions new pods and redistributes data automatically • Two SSH trust models: static keypair for dev/CI, certificate mode for production (per-cluster CA, short-lived certs, annotation-driven rotation without pod restarts) • Declarative

postgresql.conf

and

pg_hba.conf

management via

DatabaseConfig

• High availability out of the box: standby coordinator (WAL streaming, auto-promoted on failure) and mirror segments (one mirror per primary, auto-recovered via

gprecoverseg

) • On-demand backups to S3-compatible storage via

gpbackup

, with cross-cluster restore support • Phase-driven lifecycle:

PodsStarting

→

SSHReady

→

Initializing

→

Running

, with the operator reconciling continuously from there • Admission webhook that catches errors at the API boundary — cluster name length, HA node availability, invalid CRD references — before anything reaches a pod • Helm install, cert-manager for webhook TLS, Kubernetes 1.29+ Why I built it: Standing up a Cloudberry cluster is operationally heavy — SSH trust, gpinitsystem, segment registration, mirror setup, replication wiring. Keldon automates all of it so you can go from zero to a running MPP cluster without touching any of it manually. What I'm looking for: 1. Are there use cases or deployment patterns in the community I haven't accounted for? 2. Anything in the way Keldon models clusters that conflicts with how you think about Cloudberry architecture? 3. If you get a chance to try it, I'd love to hear what breaks — missing features / bug to fix Repo: github.com/keldonio/keldon-operator Docs: keldon.io/docs Happy to answer questions or take direction on anything missing. — Imad apache/cloudberry

🎉Congratulations to Liu Shengsong on becoming an Apache Cloudberry Committer! Welcome aboard! cloudberry.apache.org/blog/welcoming-liushengsong-as-a-new-apache-cloudberry-committer

🎥 The recording of our latest Apache Cloudberry Bi-weekly Community Meeting (July 10, 2026) is now available! Highlights from this meeting: • 🎉 Welcomed our new Committer Liu Shengsong • 🚀 Discussed Cloudberry 2.2 release preparation and PostgreSQL 16 stabilization • 🌐 Shared updates on cloudberry-pxf, cloudberry-backup, and Kubernetes Operator projects • 🛠️ Talked about improving release workflows, CI/CD, and documentation Thanks to everyone who joined the discussion and contributed! Watch the recording here: Feel free to share your feedback and join the ongoing discussions on our mailing lists and community channels. 🙌

🚀 Apache Cloudberry ecosystem keeps growing! We’re happy to share that Odyssey 1.5.1 has added support for Apache Cloudberry. This release improves PostgreSQL-compatible connection pooling with: • Extended protocol enhancements • Better pipelining performance • Pool balancing improvements • New metrics and observability features Welcome Odyssey to the Cloudberry ecosystem! 🎉 See more: postgresql.org/about/news/odyssey-151-released-3348

📢 Apache Cloudberry @ Community Over Code Asia 2026 If you'll be attending Community Over Code Asia in Beijing, come say hello! We'll be presenting multiple technical sessions, joining BoF discussions, hosting a community meetup, and welcoming everyone at the Apache Cloudberry booth. Whether you're already using Cloudberry or just curious about the project, we'd love to meet you and hear your ideas. 👉 Full schedule: cloudberry.apache.org/blog/apache-cloudberry-at-community-over-code-asia-2026 See you there! 👋

cloudberry 2.1 require libxerces-c-3.3.so but on readhat linux 9 libxerces-c-3.2 is available how can resolve this issue during installation of rpm

📺 Apache Cloudberry Bi-weekly Community Meeting Recording is Available Missed our latest community meeting? The recording is now available! In this meeting, we covered: • 🚀 Release and development updates • 🐘 Ongoing engineering work • 🌱 Ecosystem and community updates Watch the recording: Thanks to everyone who joined the discussion! We'd love to see more community members at future meetings.

#1875 Slow Data Insertion in Cloud Berry New discussion created by TanmayK2000 Hi, I am using Apache CLoudberry Datawarehouse for data insertion with the help of Apache Nifi i noticed that data insertion in Datawarehouse is very slow for eg i am transfering 17 million data from one table to other table which in Datawarehouse the insertion process is taking place from last 3 days and till now only 8 million data is transfer why insertion is slow any idea ?? apache/cloudberry

Hi All, What is the best approach to migrate data from greenplum 6.21 to Cloudberry 2.1? Can we get gpbackup and gprestore utilities rpm files? Thanks

#1877 why incubating the greenplum database again? New discussion created by sunnn Im intrigued to know why the incubation is happening for the same databse again as i remember there was hawq which was retired from apache long ago and also nowadays people dont even know about greenplum as many databases sucha as doris are available apache/cloudberry

Comment on #1877 why incubating the greenplum database again? Discussion answered by jianlirong Thanks for the question. The motivation behind Apache Cloudberry comes from several observations about the evolution of the PostgreSQL ecosystem over the past few years. First, Greenplum has accumulated many years of production experience and has been widely adopted across different industries. It also established itself as one of the most mature open-source MPP databases built on PostgreSQL. As the project evolved away from its previous open development model, we felt there was still a strong need in the community for a vendor-neutral, Apache-governed open-source successor that could continue serving users who value openness, long-term sustainability, and community-driven innovation. Second, Apache HAWQ and Apache Cloudberry were created to solve very different problems. HAWQ was designed as a SQL-on-Hadoop engine, with its architecture tightly integrated with the Hadoop ecosystem and based on an older PostgreSQL codebase. Cloudberry, on the other hand, is a distributed MPP database designed for modern analytics workloads, while continuously evolving together with newer PostgreSQL releases. Although the projects share some historical roots, their technical goals and target use cases are fundamentally different. Finally, the analytical database ecosystem has become much richer over the last decade. Excellent systems such as ClickHouse, Doris, and StarRocks have each established their own strengths and communities. At the same time, we continue to see strong demand for an open-source distributed database that combines: 1. close compatibility with modern PostgreSQL, 2. a proven shared-nothing MPP architecture, 3. rich SQL capabilities, 4. enterprise-grade stability and scalability, and 5. an open, community-governed development model. Apache Cloudberry is our effort to meet that demand. Rather than replacing existing systems, we hope to provide another strong option for users and organizations whose requirements align with these goals. Our goal is not to compete with any particular project, but to contribute another healthy and sustainable choice to the open-source ecosystem. We believe that diversity, open governance, and collaboration ultimately benefit both users and the PostgreSQL community as a whole. Lirong apache/cloudberry

#1886 Add LZO compression support to gpfdist New discussion created by ZTE-EBASE Description This PR adds LZO compressed file format read/write support to the

gpfdist

utility in Apache Cloudberry. Key Insight: LZO is specifically designed for speed over compression ratio. It's ideal for scenarios where decompression throughput matters more than storage savings - such as large-scale data loading into MPP databases like Cloudberry. Core Features Read Support (.lzo → Cloudberry): • ✅ Block-level decompression: Processes LZO data block by block, extracting uncompressed length and compressed length from each block header • ✅ Checksum verification: Validates data integrity using Adler32 and CRC32 checksums embedded in LZO blocks • ✅ Enhanced error reporting: Provides detailed, actionable error messages for various failure scenarios (truncated files, corrupt data, invalid format, checksum mismatches) Write Support (Cloudberry → .lzo): • ✅ Table data unloading: Exports table data to LZO-compressed external files • ✅ Standard LZOP format output: Generates files compatible with standard lzop tool and Hadoop LZO readers • ✅ Configurable compression level: Supports LZO compression levels (default: level 1, fastest; configurable up to level 9) Future Potential - Hadoop Ecosystem Integration: • 🔮 Direct query of Hive tables stored in LZO-compressed format on HDFS • 🔮 Seamless integration with Spark LZO Parquet output • 🔮 Accelerated data loading from Sqoop LZO imports Use case/motivation Scenario 1: High-Volume Data Loading LZO is a lightweight compression algorithm optimized for decompression speed (~500MB/s), making it ideal for scenarios requiring rapid data ingestion. Compared to BZip2 (~10MB/s) and GZIP (~100MB/s), LZO provides the fastest decompression while maintaining a reasonable compression ratio. Scenario 2: Hadoop Ecosystem Integration Many Hadoop ecosystem tools natively support LZO compression: 1. Hive: Supports LZO-compressed text and sequence files 2. Spark: Can write LZO-compressed output (via hadoop-lzo library) 3. Sqoop: Commonly uses LZO for bulk imports from RDBMS to HDFS References Technical Documentation • LZO Official Website • LZOP File Format Specification • Hadoop LZO Compression Guide Cloudberry Documentation • GPFDIST Official Docs • External Tables Guide Related Projects • Apache Cloudberry GitHub • Greenplum Database (upstream) Related issues No response Are you willing to submit a PR? • Yes I am willing to submit a PR! apache/cloudberry

📣 Apache Cloudberry August 2026 Incubation Report is out! <!everyone> Our latest incubation report covers the major progress from May through July 2026. Some highlights: • 🚀 PostgreSQL 16 kernel upgrade has been merged into

main

• 📦 Apache Cloudberry 2.2.0 has entered the release process • 🧊 Iceberg support has started in

datalake_fdw

• 🔧 Continued improvements to ORCA, planner, extensions, CI, security, and packaging • ☁️ Ecosystem projects including PXF, cloudberry-backup, cloudberry-go-libs, WAL-G, and Kubernetes operators continued to move forward • 👥 Anton Kurochkin and Liu Shengsong joined as new Committers • 🌏 Community engagement continued through Community Over Code Asia 2026 and other events Thanks to everyone who continues to contribute to Cloudberry — whether through code, reviews, documentation, testing, releases, or community discussions! 🙌 🫱🏻 Read the full incubation report: cloudberry.apache.org/blog/apache-cloudberry-incubation-report-202608

The recording of our latest Apache Cloudberry (Incubating) bi-weekly community meeting is available: Highlights include: • Recent PRs and feature discussions • Release-branch management and cherry-picking • Behave test automation and CI • Rocky Linux 10 support and release progress • Cloudberry ecosystem and website updates • Community events and Kubernetes operator collaboration Please watch the recording and share your thoughts with the community!

We’ve launched a new Size Calculator for Apache Cloudberry to help with hardware planning before starting a POC. Enter your uncompressed data size, choose a physical, VM, or cloud deployment, and get an estimated configuration covering nodes, CPU, memory, storage, networking, and primary/mirror layout. Advanced options also allow you to adjust compression ratio and query concurrency. It’s designed to provide a practical starting point for POC planning—not a replacement for workload validation. Try it here: cloudberry.apache.org/sizing Many thanks to @Ryan Wei for providing the prototype and the original idea!

#1939 Apache Cloudberry through ODBC into Apache Arrow (adbcBridge 0.1.0) — is multi-argument unnest a safe bulk-ingest path on Cloudberry? New discussion created by singhpratech adbcBridge is a small Apache-2.0 ADBC driver written in C11 that loads an ODBC driver and exposes it through the Arrow ADBC C ABI: block-cursor reads straight into Arrow record batches, bulk ingest, metadata, partitioned parallel reads. I ran Apache Cloudberry through it as one of 46 databases in a single compatibility workload, and it is one of the few entries that needed no driver quirks and no tolerance flags at all — so I wanted to share the entry here and ask one question at the end. What was verified (Apache Cloudberry 2.1.0-incubating,

woblerr/cloudberry

image; driver psqlodbc 16, PostgreSQL wire): • Linux: PASS • macOS arm64: PASS (amd64 emulated) • Windows x64: PASS (compose service unchanged, 3 GB / shm 1 GB) What the compatibility entry records: an MPP cluster of PostgreSQL 14 segments behind one coordinator, driven by the

postgres

entry's types unchanged (

INTEGER

,

DOUBLE PRECISION

,

VARCHAR

,

BYTEA

,

DATE

,

TIMESTAMP

,

NUMERIC(10,3)

,

BOOLEAN

all behave as on stock PostgreSQL) and, unlike CockroachDB, needing no

PRIMARY KEY

. Since it reports

SQL_DBMS_NAME

"PostgreSQL" behind the same

psqlodbcw.so

, no driver-name quirk could be correct here without also firing on real PostgreSQL; the only place Cloudberry is named in the bridge is a fork test on the

version()

banner. Extra steps cover what the standard workload cannot tell apart from PostgreSQL: a

DISTRIBUTED BY

table whose bulk-ingested rows occupy both segments plus an aggregate merged on the coordinator (

Gather Motion 2:1

, GPORCA), and append-optimized column-oriented storage read from

pg_am

as

ao_column

. Full entry: https://github.com/singhpratech/adbcbridge/blob/main/docs/COMPATIBILITY.md The question. For real PostgreSQL the bridge ingests through a multi-argument

unnest

form —

INSERT INTO t SELECT * FROM unnest($1::bigint[], $2::text[], …)

with one array parameter per column — and it only turns that on when

version()

is a PostgreSQL banner carrying no fork marker, so Cloudberry (banner

PostgreSQL 14.4 (Apache Cloudberry 2.1.0-incubating …)

) keeps the plain multi-row

INSERT

path. Probing Cloudberry directly, the

unnest

form works exactly as on PostgreSQL — from array literals and from bound `bigint[]`/`text[]` parameters, into heap, append-optimized row and append-optimized column tables, across the segments — and server-side it lands 5,000 rows roughly an order of magnitude faster than the multi-row

INSERT

it keeps (~530k vs ~33k rows/s, bare SQL on a shared host). Is relying on multi-argument

unnest

expansion something Cloudberry considers stable across releases, the same way PostgreSQL does — i.e. is it reasonable to let Cloudberry through that gate? If there is a caveat (GPORCA vs the planner, distribution of the unnested rows, anything on the 2.x roadmap), I would rather know before flipping it. Trying it (Python; Rust, Go, Java and C# are on the docs site):

pip install adbcbridge

import adbcbridge with adbcbridge.connect(uri="Driver=psqlodbcw.so;Servername=127.0.0.1;Port=5432;Database=mydb;Username=gpadmin;Password=…;") as conn: with conn.cursor() as cur: cur.execute("SELECT ...") table = cur.fetch_arrow_table() # a pyarrow.Table Links: repository https://github.com/singhpratech/adbcbridge · docs https://adbcbridge.org/docs/ · upstream notes https://github.com/singhpratech/adbcbridge/blob/main/docs/UPSTREAM.md · PyPI https://pypi.org/project/adbcbridge/ It is a 0.1.0. If the entry says something wrong about Cloudberry, or you run a version I didn't, an issue on the repository with the details is the most useful thing you could send. apache/cloudberry

Curious about how Apache Cloudberry compares with Greenplum? Cloudberry keeps the familiar Greenplum experience—most SQL syntax, utilities, and operational workflows remain compatible—while adding a modern PostgreSQL kernel and new capabilities across storage, performance, security, deployment, and operations. Highlights include PAX Storage Format, Directory Tables, incremental materialized views, runtime filters, AO table indexes, TDE, row-level security, single-node deployment, ARM support, and more. It is also governed openly under The Apache Way, with contributions and decisions driven by the community. Read the full overview: cloudberry.apache.org/docs/introduction/cbdb-vs-gp-features

Hi everyone, sharing a new Apache Cloudberry ecosystem update: BladePipe 1.6.0+ now supports Apache Cloudberry, enabling visual, low-code data migration and synchronization from PostgreSQL and Greenplum. The integration supports: • Schema migration • Full data migration • Incremental data synchronization • Data verification and correction • Common DDL synchronization Step-by-step guide: cloudberry.apache.org/docs/ecosystem/data-integration/bladepipe Original news: lists.apache.org/thread/… Community feedback on compatibility, usability, and real-world scenarios is very welcome.

#1943 [Ideas] data encrypt or privacy infomation protection New discussion created by yz271544 Description The following project describes a data encryption/decryption algorithm previously implemented in the Teradata data warehouse, which has also been used on other platforms. The current Cloudberry could consider adding the issuance of decryption JWT tokens to the console, for example, by adding an administrator-set key to the claim. Then, the JWT token could be configured at the JDBC or command-line connection session level to support encryption and decryption in various scenarios. It's also worth considering scenarios where data is encrypted during loading and decrypted during export. Alternatively, it could support extended interfaces like Teradata's fastload and fastexp, providing inmod and outmod functions for both data loading and export. data encrypt and decrypt demo Use case/motivation The goal is to enable native support for privacy data protection scenarios in next-generation data warehouses. Related issues No response Are you willing to submit a PR? • Yes I am willing to submit a PR! apache/cloudberry

#1952 [Proposal] Continuous SQLancer fuzzing for Cloudberry (5 bugs from a first run, looking for volunteers) New discussion created by my-ship-it Proposers @roseduan, @my-ship-it Proposal Status Under Discussion Abstract Hi all, @roseduan and I recently pointed SQLancer at Cloudberry

main

to see what would happen. Using the stock PostgreSQL provider and no Cloudberry-specific tuning, a few hours of fuzzing surfaced 5 distinct bugs: one silent wrong result, two internal ERRORs, one backend segfault, and one assertion failure. Rose has fix PRs up for two of them, and the other three are filed as issues below. Given how little effort this took, we think Cloudberry would benefit from running SQLancer regularly and automatically. SQLancer is already listed as a planned testing item on the project roadmap (#868), and it has paid off here before: @shmiwy found #594 and #596 with it in 2024, both since fixed (#722, #598), and @congxuebin's #317 in 2023 was a useful data point even though it turned out to be expected behaviour. What we are proposing is to make this routine instead of an occasional one-off. To be upfront: neither of us can drive the automation work ourselves right now. We are posting this to share what we found, sketch a plan that we think is realistic, and see whether a few people would like to pick up pieces of it, with our help getting started. What we found | # | Component | Symptom | Status | | - | ---------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------- | | 1 | ORCA | SELECT DISTINCT / GROUP BY on a nullable UNIQUE column returns duplicate NULLs (silent wrong result) | PR [#1941](https://github.com/apache/cloudberry/pull/1941) | | 2 | ORCA | Merge FULL JOIN whose one side is provably empty (e.g. a partitioned table with no partitions): ERROR: unexpected gang size | PR [#1896](https://github.com/apache/cloudberry/pull/1896) | | 3 | ORCA | Filter on an INCLUDE-only index column is pushed into the Index Cond: ERROR: bogus index qualification | [#1948](https://github.com/apache/cloudberry/issues/1948) | | 4 | ORCA | QD backend segfault (null dereference) in extended-statistics cardinality estimation when a dependencies statistics object does not cover all filtered columns | [#1949](https://github.com/apache/cloudberry/issues/1949) | | 5 | Planner (Cloudberry-specific code) | Pushed-down OR selectivity outside [0, 1] under a LEFT JOIN trips the assertion in adjust_selectivity_for_nulltest() (assert-enabled build) | [#1950](https://github.com/apache/cloudberry/issues/1950) | All five reproduce on current

main

with a 3-segment demo cluster built with

--enable-cassert

. The run used the stock

postgres

provider against a 3-segment demo cluster built with

--enable-cassert

. The

optimizer=on

vs

off

comparison that exposed bug #1 was done by hand on the failing queries; SQLancer itself does not do that yet (see Implementation). Reproducers 1. ORCA drops the Agg for DISTINCT over a nullable UNIQUE column create table repro (c0 numeric unique); insert into repro values (1), (2), (null), (null), (null); select distinct c0 from repro; -- optimizer=on -> 5 rows (plan is Gather Motion -> Seq Scan, no aggregate) -- optimizer=off -> 3 rows 2. Merge FULL JOIN with a provably-empty side CREATE TABLE gs_part (a int) PARTITION BY RANGE (a) DISTRIBUTED BY (a); CREATE TABLE gs_r (a int) DISTRIBUTED BY (a); CREATE TABLE gs_s (a int) DISTRIBUTED BY (a); SET optimizer = on; EXPLAIN SELECT * FROM gs_s, gs_r FULL JOIN gs_part ON gs_r.a = gs_part.a; -- ERROR: unexpected gang size: 3 (nodeMotion.c) 3. Bogus index qualification with an INCLUDE column CREATE TABLE bogus_t (c0 boolean, c1 boolean); CREATE INDEX bogus_i ON bogus_t (c0) INCLUDE (c1); INSERT INTO bogus_t VALUES (true, true), (false, true), (true, false); SET optimizer = on; SELECT * FROM bogus_t WHERE c1; -- ERROR: bogus index qualification (nodeIndexscan.c) -- EXPLAIN shows "Index Cond: (c1 = true)" on bogus_i; c1 is not a key column 4. QD segfault when extended statistics do not cover all filtered columns CREATE TABLE t3 (c0 boolean, c1 text, c2 int) DISTRIBUTED BY (c0); INSERT INTO t3 SELECT (g%2=0), 'x'||g, g FROM generate_series(1,100) g; CREATE STATISTICS s0 (dependencies) ON c0, c1 FROM t3; -- covers c0, c1 only ANALYZE t3; SET optimizer = on; SELECT * FROM (SELECT ALL t3.c0 AS t3c0, t3.c1 AS t3c1, t3.c2 AS t3c2 FROM t3 WHERE (t3.c0) IS TRUE GROUP BY t3.c0, t3.c1, t3.c2 ORDER BY t3.c1) AS result WHERE result.t3c0 = TRUE AND result.t3c1 = '' AND result.t3c2 > 0; -- c2 not covered -- SIGSEGV in CExtendedStatsProcessor::ApplyCorrelatedStatsToScaleFactorFilterCalculation -- optimizer=off returns 0 rows 5. Selectivity outside [0, 1] under an outer join (assert build) CREATE TABLE m1(c0 inet); CREATE TABLE m2(c0 inet); INSERT INTO m2 VALUES ('88.147.138.141'), ('76.163.212.11'), ('214.10.65.144'); ANALYZE m1, m2; SELECT COUNT(*) FROM ONLY m1 LEFT OUTER JOIN m2 ON true WHERE (m1.c0 IS NOT NULL) OR (m2.c0 BETWEEN SYMMETRIC '75.175.243.19' AND '230.9.216.68'); -- FailedAssertion("pselec >= 0.0 && pselec <= 1.0", costsize.c) -- adjust_selectivity_for_nulltest() is Cloudberry code (gp_adjust_selectivity_for_outerjoins); -- upstream PostgreSQL has no equivalent assertion, a non-assert build just gets a bad estimate Try it yourself in about 10 minutes # Cloudberry with asserts + a demo cluster on port 7000 ./configure --enable-cassert --enable-debug <your usual flags> && make -j$(nproc) install make create-demo-cluster && source gpAux/gpdemo/gpdemo-env.sh createdb test # the postgres provider connects to "test" first # SQLancer (no recent Maven Central release, so build from main) git clone --depth 1 https://github.com/sqlancer/sqlancer && cd sqlancer && mvn -q package -DskipTests java -jar target/sqlancer-*.jar --num-threads 4 --num-queries 1000 --timeout-seconds 3600 \ --username gpadmin --password '' \ postgres --connection-url postgresql://localhost:7000/test \ --oracle NOREC --test-tablespaces false --test-collations false # also try --oracle QUERY_PARTITIONING (TLP). Failing cases land in ./logs/postgres/<db>.log # as a replayable script with the random seed in the header. For each one, re-run the # final query with optimizer=on and optimizer=off and compare. Motivation • Bug #1 is the kind a hand-written regression suite is not designed to catch. No error, no crash, just a different answer than the Postgres planner gives. SQLancer's logic-bug oracles (TLP, NoREC, and PQS) exist for exactly this class of problem. • Cloudberry has a differential oracle most databases do not have. Any query where

optimizer=on

and

optimizer=off

disagree is a bug in one of them. SQLancer calls this idea Differential Que… apache/cloudberry

Comment on #1943 [Ideas] data encrypt or privacy infomation protection Discussion answered by pavetheroad41 Thanks for putting this together - data protection is a real gap for a lot of enterprise Cloudberry/Greenplum deployments, so it's good to see this raised. A few thoughts and questions on the proposal: Scope and threat model It would help to pin down what threat model this is targeting - encryption at rest (protecting data files/backups if storage is compromised), encryption in transit (already largely covered by TLS on JDBC/libpq), or column-level/application-layer encryption (protecting specific sensitive fields even from DB admins). The JWT + claims approach you're describing sounds closest to the third case, which has very different design constraints than TDE-style at-rest encryption. Prior art to consider pgcrypto already gives Postgres-family databases column-level encrypt/decrypt functions (pgp_sym_encrypt, etc.). Worth clarifying how this proposal differs - is the goal mainly to move key/claim management into a JWT-based session context rather than passing keys explicitly in SQL? Greenplum/Cloudberry's MPP architecture means encryption/decryption work needs to be pushed down to segments efficiently, not just the coordinator, this could get expensive at scale if not carefully implemented (e.g., per-row UDF calls). JWT-based key delivery Using a JWT claim to carry a decryption key (or a wrapped key) at session level is an interesting idea, but it raises questions: How would key rotation work? What's the failure mode if a token expires mid-session or mid-COPY? Would the key material touch server logs or EXPLAIN output anywhere? Load/export integration The fastload/fastexp-style inmod/outmod hooks are a reasonable model - Cloudberry's gpfdist/external table framework already has extensibility points that might map naturally onto this (custom protocol handlers or format functions) rather than requiring new machinery. Happy to help think through a design doc if there's interest in moving this forward, particularly around how this would interact with the segment/coordinator split and existing external table protocols. apache/cloudberry

Hi all,   I'm running into a recurring, intermittent failure during gpinitsystem segment initialization on Cloudberry DB 2.1.0.   Environment: - Cloudberry DB 2.1.0 - 32 primary segments total (8 per host on all hosts)   Issue: During gpinitsystem's parallel segment creation, a small number of segments fail at the "Configuring segment pg_hba.conf" step with an error like:   gpcreateseg.sh&lt;host&gt;gpadmin-[FATAL][<seg_id>]:-Failed Update pg_hba.conf for coordinator IP address fe80::<link-local-address>/128   This has happened across multiple attempts, each time on different segments (initially gpseg17 and gpseg23, most recently gpseg19), so it isn't tied to one specific segment or host — it looks like an intermittent race during the parallel pg_hba.conf update step.   What we found and already fixed: gpinitsystem enumerates every IPv6 address on the coordinator, including link-local (fe80::) addresses, and writes a trust entry for each into every segment's pg_hba.conf. We found one of these link-local addresses belonged to the coordinator's "idrac" interface (Dell iDRAC virtual NIC, unrelated to the cluster network). We disabled IPv6 on that interface (net.ipv6.conf.idrac.disable_ipv6=1) to remove it from the enumerated address list.   After that fix, gpinitsystem got further, but a new segment (gpseg19) still failed with the same style of error, again on a link-local IPv6 address during the pg_hba.conf update step.   Questions: 1. Is this a known issue with how gpinitsystem/gpcreateseg.sh enumerates and writes trust entries for coordinator link-local IPv6 addresses during parallel segment creation? 2. Is there a way to exclude link-local (fe80::/10) addresses entirely from the pg_hba.conf population, rather than tracking down and disabling IPv6 per interface? 3. Could this be a race condition in the parallel pg_hba.conf update logic itself (multiple gpcreateseg.sh processes hitting the same coordinator address list concurrently), rather than purely a network/interface issue?   Happy to share the full gpinitsystem log if useful. Any guidance on the right fix (network-side vs. a gpinitsystem/gpcreateseg.sh behavior) would be appreciated.   Thanks!

#1958 [Ideas] Add instrumentation and latency metrics to the Anser subsystem New discussion created by leborchuk Description Description Anser (

gpcontrib/anser/

) is a runtime pub/sub facility: producer nodes on the segments publish a bloom filter over a join-build key, the coordinator unions the parts, and consumer nodes on the segments receive it and prune probe rows. See

gpcontrib/anser/README.md

for the architecture before starting. Today we can see what the filter did (

Rows Removed by Bloom Filter

in

EXPLAIN ANALYZE

) but nothing about what it cost. A producer blocks until the coordinator acknowledges its part; a consumer blocks until the payload is delivered. If those waits are expensive, the filter can be a net loss and we currently have no way to tell. Goal Make every wait and every queueing delay in Anser measurable, both per query (in

EXPLAIN

) and cluster-wide (in shared memory). Definition of done •

EXPLAIN (ANALYZE, VERBOSE)

shows producer and consumer wait count and wait time, including for segment-executed nodes. •

SELECT * FROM anser.stats()

shows queue time, service-cycle latency, and the histograms;

anser.stats_reset()

works. •

SELECT * FROM anser.session_stats()

reflects the current session only. • Regression tests pass and contain no timing-dependent expected values. •

README.md

gains a "Metrics" section: every field, its unit, and where it is measured. • No new LWLock acquisitions on the data path; no per-tuple timing calls. • Core (

src/

) is untouched. What to measure There are two distinct vantage points. Do not mix them up — this is the single most important thing to understand before writing code. | Vantage point | Who waits | Where the wait actually happens | | ---------------------------------------- | ---------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | Executor node (segment or coordinator) | the query | segment: the libpq round trip in AnserClientPublish / AnserClientConsumeWait (src/anserclient.c); coordinator: the direct calls AnserPublish / AnserWaitProducersRegistered + AnserWaitReady + AnserConsumeReady (src/anser.c) | | Coordinator internals | the QD backend serving a segment, and the two background workers | AnserWaitSubmissionAck (src/anser.c:906), AnserWaitSlotResult (src/anser.c:1110), AnserGatherServiceCycle (src/anser.c:1235), AnserSendServiceCycle (src/anser.c:1433) | The first belongs in

EXPLAIN

. The second belongs in shared-memory counters, because no single backend sees it. Implementation Suggest three changes in this order. Each is independently reviewable and testable. Change 1 — per-node wait time in EXPLAIN • Time the publish call in the producer node and the receive call in the consumer node. Record, per node: number of waits, total wait time, and longest single wait. • Producer:

AnserProducePublishPart()

in

src/anserbloomproduce.c

— it picks the transport by role, so wrapping it covers both segment and coordinator. • Consumer:

ExecAnserBloomFilterConsume()

in

src/anserbloomconsume.c

. • Store the accumulators in

AnserBloomProduceScanState

/

AnserBloomConsumeScanState

(

src/anserplanexec.c:81

and

:95

). • Print them from

anser_produce_explain()

(

src/anserplanexec.c:422

) and

anser_consume_explain()

(

:636

), gated on

es->analyze

(there is nothing to print without execution) and shown only at

EXPLAIN (ANALYZE, VERBOSE)

— i.e.

es->analyze && es->verbose

. Use

ExplainPropertyFloat("Anser Wait Time", "ms", ...)

and

ExplainPropertyInteger("Anser Waits", NULL, ...)

. The trap you must handle. A field you add to the node state on a segment does not reach the QD. Read the comment at `src/anserplanexec.c:583-588`: the existing code deliberately uses

InstrCountFiltered1/2

because only the fixed fields of

CdbExplain_StatInst

(

src/backend/commands/explain_gp.c:44

) travel back. Your new timers are not in that struct. The supported escape hatch is the per-node extra text channel, which is how

Hash

reports `Extra Text: (seg2) Hash chain length ...`: •

PlanState.cdbexplainbuf

and

PlanState.cdbexplainfun

(

src/include/nodes/execnodes.h:1152-1153

) • collected on the segment by

cdbexplain_collectExtraText()

(

src/backend/commands/explain_gp.c:1308

) and shipped to the QD • copy the pattern from

src/backend/executor/nodeRuntimeFilter.c:174-177

— it is the closest existing analogue (a runtime-filter node doing exactly this) So: allocate

cdbexplainbuf

in

*_begin

when

estate->es_instrument

is set, install a

cdbexplainfun

that appends your numbers, and the QD will show them as

Extra Text

per segment. No core change is needed — if you find yourself editing anything under

src/

, stop and re-read this paragraph. Change 2 — subsystem-wide counters and latency histograms in shared memory • Add a stats block to Anser's shared memory. Either extend

AnserControl

(

include/anser.h

) or add a separate struct; account for it in

AnserShmemSize()

and initialize it in

AnserShmemInit()

(

src/anser.c

), exactly as the existing structures do. • Use

pg_atomic_uint64

(

src/include/storage/atomics.h

) for every counter and bucket, and update them without taking

AnserChannelLock

/

AnserRingLock

. These are on the data path; adding lock traffic for metrics would change what we are trying to measure. • Measure, at minimum: | Metric | Where | | ------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------ | | time a part waits in the submission queue (enqueue → gather picks it up) | stamp the time in AnserEnqueueSubmission() (src/anser.c:860), read it in AnserGatherServiceCycle() (:1235) — add a field to AnserSubmissionEntry | | time a consumer slot waits for delivery (register → send stamps payload) | stamp in AnserRegisterWaitSlot() (:1077), read in AnserSendServiceCycle() (:1433) — add a field to AnserWaitSlot | | producer ack wait, on the QD side | AnserWaitSubmissionAck() (:906) | | consumer delivery wait, on the QD side | AnserWaitSlotResult() (:1110) | | background worker cycle duration, per service | around the AnserGatherServiceCycle() / AnserSendServiceCycle() calls in AnserServiceLoop() (src/anserservice.c) | | counts: parts submitted / accepted / rejected, channels cancelled by timeout, consumers delivered / cancelled | the same call sites … apache/cloudberry

#1959 [Ideas] Bloom filter performance: four experiments New discussion created by leborchuk Description This is a research issue. The deliverable for each task is a short report with numbers and a recommendation — not a patch. Code is welcome as evidence, but "we measured X, therefore do Y" is the output we want. A negative result is a successful outcome and must be reported with the same rigour as a positive one: "we tried this, it gained 0.8%, not worth the complexity" is valuable and closes the question permanently. Each of the four tasks below is independent and can be a separate PR/report. Read "How to produce results we can act on" before starting any of them. The current implementation, in facts | Aspect | Anser today | Where | | ------------------ | ---------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------- | | sizing target | 2 bytes (16 bits) per key | bloom_create(), src/backend/lib/bloomfilter.c:96-101 | | bitset floor / cap | 1 MB floor, 64 MB cap | same, plus ANSER_RF_MIN_BYTES / ANSER_RF_MAX_BYTES in gpcontrib/anser/src/anserplan.c:50-51 | | bitset rounded | down to a power of two | my_bloom_power() | | hash functions | optimal_k() = round(0.693 × m/n), capped at 10 | bloomfilter.c:MAX_HASH_FUNCS, optimal_k() | | per-element cost | one hash_any_extended() + k cheap derivations + k random bitset accesses | k_hashes() | | union of parts | for (i...) abits[i] \|= pbits[i]; — one byte at a time | AnserBloomFoldPartInPlace(), gpcontrib/anser/src/anserfilter.c | | union lock | runs while holding AnserChannelLock EXCLUSIVE | AnserStorePayloadDSM(), gpcontrib/anser/src/anser.c:1868 (the Assert) | | wire format | raw header + raw bitset, uncompressed, binary bytea | AnserBloomSerializePart(), sent by anserclient.c | | give up early | never | — | Cloudberry already answers three of these questions differently The core runtime filter (

src/backend/executor/nodeRuntimeFilter.c

) uses

bloom_create_aggresive()

(

bloomfilter.c

), and its choices are the opposite of Anser's: | Anser (bloom_create) | core RF (bloom_create_aggresive) | | | -------------------- | -------------------------------- | ---------------------------------------------------------------------------------------------------------- | | bits per key | 16 | 9 (total_elems * 9 / 8 bytes) | | floor / cap | 1 MB / 64 MB | 128 KB / 2 MB | | hash functions | up to 10 (optimal_k) | 2 or 3, hard-coded, "balance false positive rate and calculation costs" | | refuses to build | never | when bits_per_elem < 1.6 → returns NULL, "too many elements" | | abandons mid-build | never | yes: inner_threshold = bloom_total_bits(bf) / 1.6, then build_suspend = true (nodeRuntimeFilter.c:246,311) | Start every task by explaining why Anser differs from this, and treat the aggressive profile as a candidate answer that someone in this codebase already committed to. Do not invent new thresholds before measuring against that one. How to produce results we can act on This section is the actual hard part of this issue. Results that don't follow it will be sent back, not because of process for its own sake, but because we cannot make a shipping decision from them. 1. Build discipline (most common way to waste a week) • Never benchmark a build configured with

--enable-cassert

or

--enable-debug

only. Assertions make PostgreSQL 20-40% slower and distort exactly the kind of tight loop you are measuring. Configure a separate optimized build (

-O2

, no cassert) and paste the

configure

line into your report. • Record the compiler and version. Auto-vectorization differs between GCC and Clang and between versions; a result without a compiler version is not reproducible. • Pin a baseline commit hash and re-measure the baseline in the same session as the variant. Never compare against numbers from yesterday's machine state. 2. Measure at two layers, always both | Layer | How | Answers | | ------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ---------------------------- | | Micro | a SQL-callable C benchmark added to gpcontrib/anser/src/anser_test.c (follow the existing anser_test_bloom_* helpers), e.g. anser_test_bench_fold(bytes, iters) returning elapsed ms | "is the OR faster?" | | End-to-end | query wall time on a demo/real cluster, TPC-DS with SF=100 | "does the query get faster?" | A micro-benchmark win that does not move end-to-end time is a negative result — report it as such. That happens constantly and is fine. Keep benchmark helpers out of the regression expected output; they are nondeterministic. They are for manual runs. Notes about TPC-DS Scale factor 100 minimum. SF 1 and SF 10 are not acceptable evidence: the tables fit in memory, the filter is pure overhead, and the result tells us nothing about production. Use SF 1000 if the hardware allows, and state the cluster shape (hosts × segments per host). You could use special extension to run TPC-DS https://github.com/avamingli/pg_tpcds Step 1 — find out which queries Anser even touches. The plan pass only injects into one join shape, so most of the 99 queries will be untouched and their timings are noise. Establish the injected subset first: Report that list — it is a result in its own right, and if it is empty, stop and open a bug instead of benchmarking. Expect the star-schema fact/dimension joins (q3, q7, q13, q17, q19, q25, q29, q42, q43, q46, q52, q55, q64, q68, q72, q79, q88, q96, q98 are the usual candidates), but use your measured list, not this one. Step 2 — run the full suite, 3 iterations, `anser.runtime_fi on, everything else identical. Report per-query medians. Step 3 — report in three parts: 1. Injected queries — the detail table: | Query | baseline (s) | variant (s) | Δ% | consumer wait (ms) | | --- | --- | --- | --- | --- | --- | | q19 | | | | | | 1. Non-injected queries — one line: worst regression observed, of que… apache/cloudberry

#1962 [Ideas] Adaptive join: correct a wrong Broadcast decision at runtime New discussion created by leborchuk Description The planner picks

Broadcast Motion

for the inner side when it estimates the inner side is small. When that estimate is wrong by 10× or 100×, every segment receives all inner rows and builds the entire hash table — N× the network traffic, N× the build CPU, N× the memory, and often a spill to disk that redistributing would have avoided. The query does not fail; it just runs far slower than the alternative plan, and today nothing notices. Anser could be used to switch from broadcast to redistribute (or vice versa - to discuss). The user-facing requirement, which drives the whole design: boundary must be a region where one algorithm is unambiguously better. Not "they are within 10% and the trend favours one" — a zone where obviously right, with a deliberate no-switch band around it. What we can reuse Anser's channel payload is opaque bytes; only the fold is bloom-specific (

AnserBloomFoldPartInPlace()

checks the

ABF1

magic in

gpcontrib/anser/src/anserfilter.c

). Statistics fold just as easily: | Payload | Fold operation | Use | | ---------------------- | ----------------------- | ----------------------- | | bloom filter (today) | bitwise OR | pruning | | row count / byte count | integer addition | this issue, milestone 1 | | Count-Min sketch | element-wise addition | skew (milestone 3) | | Space-Saving top-K | mergeable summary union | skew (milestone 3) | So milestone 1 needs a second payload type with its own magic a not a new transport. Design that generalization deliberately — a payload-type tag in the header plus a fold callback — because milestone 3 de Milestone 0 — feasibility and correctness (do this first, do not skip) Before any code or benchmarks, answer this in a design note. It is entirely possible that the honest answer to part of this issue is "not feasible without planner changes", and finding that out in week 1 is a success. Facts to start from: •

MOTIONTYPE_HASH

vs

MOTIONTYPE_BROADCAST

(

src/include/nodes/plannodes.h:1880-1886

) differ, at the sender, only in how each tuple's destination is chosen. The receiving slice is the same set of processes either way, and the gang is already running. So flipping the sender's routing is not obviously impossible. • But the join above the motion is only correct if both inputs are placed compatibly. Broadcast-inner works with the outer left wherever it is. Redistribute-inner requires the outer to also be hashed on the join key. If the plan broadcast the inner precisely so the outer would not need a motion, you cannot flip one side in isolation — you would silently produce wrong results, which is far worse than being slow. That yields one case where a runtime switch is provably safe:

The outer side is already distributed on the join key (it was scanned from

a table distributed by that key, or a motion earlier in the p

it there). Then broadcast-inner and redistribute-inner are both correct, and

the choice is purely about cost.

Milestone 0 deliverable — a note answering: 1. In which plan shapes is the outer side already hash-distributed on the join key? How often does that happen in TPC-DS at SF100? (Count it — this bounds the value of the whole issue.) 2. Can all senders in a slice agree on a routing switch, and what synchronizes them? (They must agree, or tuples for the same key land on different segments and rows are lost.) 3. What happens to inner tuples already sent before the switch? Does the inner side need re-scanning, and if so what does that cost? 4. Where would the decision be taken and enforced — in

nodeMotion.c

at the sender, or by choosing between two pre-planned alternatives at slice start? 5. For the general (unsafe) case, which is more realistic: planning both alternatives and choosing at execution start, or aborting and re-dispatching with corrected cardinality (what Spark AQE does at shuffle boundaries)? Do not begin milestone 2 until a reviewer has agreed with this Milestone 1 — the cost model and where the boundary is (the core research) What the system already believes The Postgres-planner motion cost (

cdbpath_cost_motion()

,

src/backend/cdb/cdbpath.c

): cost_per_row = (gp_motion_cost_per_row > 0) ? gp_motion_cost_per_row : 2 * cpu_tuple_cost; /* = 0.02 */ motioncost = cost_per_row * 0.5 * (sendrows + recvrows); With R inner rows and N segments: redistribute has

recvrows ≈ R

, broadcast has

recvrows = R × N

. So the model says broadcast-inner beats redistributing both sides when, roughly:

0.5 · R_in · (1 + N)   <   R_in + R_out
  ⇒  R_out  >  R_in · (N − 1) / 2

i.e. at N=3 the outer must merely exceed the inner; at N=48 it must exceed it 23×. ORCA carries a separate, blunter rule:

optimizer_penalize_broadcast_threshold

= 100 000 rows by default (

guc_gp.c:4531

). Your first job is to find out whether this linear model is true. It has no term for any of the following, and at least two of them are non-linear. The four cost components — measure each | # | Component | Broadcast | Redistribute | Why it may dominate | | - | ------------------------------ | ---------------------------------------- | --------------- | -------------------------------------------- | | 1 | bytes on the wire | R × N received | R received | linear, but interconnect flow control is not | | 2 | receive-side CPU (deserialize) | R × N | R | linear | | 3 | hash table build | every segment builds all R rows | each builds R/N | N× CPU and N× memory | | 4 | spill to disk | R × width vs work_mem on every segment | (R/N) × width | step function, 2-10× | The strongest candidate for an unambiguous boundary Component 4 is a step function, and steps are exactly what "no doubt" looks like:

If broadcasting makes the hash table exceed

work_mem

and spill, while

redistributing keeps it in memory, redistribute wins by a large, verifiable

margin — regardless of the row count.

That reframes the trigger from "how many rows" to "which side of the spill boundary each alternative lands on", with the row count as the input to that test. ORCA's flat 100 000 rows is a crude proxy for the same thing that ignores row width,

work_mem

, and segment count. Verify this before building on it. Measure query time for broadcast and redistribute while sweeping inner rows across the point where broadcast starts spilling, at fixed

work_mem

. If the curve shows a sharp knee, that is your boundary and you can defend it. If it degrades smoothly, say so — the recommendation then has to come from the K-factor rule below alone. The switch rule you must propose Express it as a rule with a deliberate dead zone, and state every constant with the measurement that produced it:

switch broadcast → redistribute  iff
      cost_broadcast  >  K × cost_redistribute        (K ≥ 2, measured)
  AND estimated absolute saving > S                   (e.g. > 1 s or > 1 GB moved)
  AND cost_of_switching_now < estimated saving

• K is the no-doubt factor. Within K× we do not switch, on purpose. Justify the value you pick from the spread of your own measurements: K must be larger than your measurement noise by a comfortable margin. • S prevents churn on queries too small for any of this to matter. • The switch itself is not free — item 3 of the milestone-0 note. If the inner side must be re-scanned, the switch cost is a full inne… apache/cloudberry

#1963 [Ideas] Implement Robust Predicate Transfer (RPT+) on top of Anser New discussion created by leborchuk Description The references, precisely | What | Where | Note | | ----------------------- | -------------------------------------------------------------------------------------------- | --------------------------------------------- | | RPT paper | Junyi Zhao et. all Debunking the Myth of Join Ordering: Toward Robust SQL Analytics | https://arxiv.org/pdf/2502.15181 | | DuckDB RPT extension | https://github.com/robust-sql/robust | this is the baseline RPT | | RPT+ paper | Yiming Qiao, Peter Boncz, Huanchen Zhang, Robust Predicate Transfer with Dynamic Execution | https://vldb.org/pvldb/vol19/p1278-qiao.pdf | | DuckDB RPT+ extension | https://github.com/embryo-labs/dynamic-predicate-transfer | this is RPT+ | Read both repos, but know which is which:

robust-sql/robust

implements plain Predicate Transfer (DuckDB's native bloom filter, 12 bits/key; reports 1.76× geomean on JOB), while the paper is specifically about fixing RPT's overhead and regressions. ⚠️ Licensing. Cloudberry is Apache-2.0. Check the license of both repos before reusing even a snippet, and prefer implementing from the paper. If you do reuse anything, it needs attribution plus `LICENSE`/`NOTICE` handling — raise it in the PR rather than deciding alone. What the paper actually proposes RPT (the baseline) makes Yannakakis' algorithm practical: model the query as a join graph

G(V,E)

where edge weight = number of shared join attributes, build a maximum spanning tree rooted at the largest table (

LargestRoot

), then propagate bloom filters over that tree in a forward pass (leaves→root) and a backward pass (root→leaves), fully reducing tables before the join phase. Its weakness: it ignores equivalence relationships between join keys, producing redundant and oversized filters, and pays heavy scan + BF-construction overhead — enough to cause real regressions. RPT+ adds three things: 1. Asymmetric Transfer Plan (ATP) — different tree shapes for the two passes, because they do different jobs. Forward = collect information, so a deep chaining tree is best (filter table i before building the filter for i+1, avoiding oversized BFs). Backward = distribute information, so a wide broadcast tree is best (build the final filter once and reuse it for every table in the same equivalence class). Chains for different equivalence classes are linked through "bridge tables" (the root of each class's induced subtree). 2. Cascade filters — min/max for block-level skipping plus a bloom filter for tuple-level filtering, applied coarse-to-fine. 3. Dynamic pipelines — runtime selectivity monitoring that abandons filter construction when it will not pay off (Algorithm 2 in §5.2). Reported speedups over DuckDB v1.3.0: 1.47× JOB, 1.28× SQLStorm, 1.17× TPC-H, 1.01× Appian, "avoid[ing] the significant performance regressions observed with the original RPT". Constants from §5.5, use these as starting values, do not invent your own: | Parameter | Value | Meaning | | ---------- | ------------------------------- | ----------------------------------------------------- | | γ | 100 K tuples | rows observed before deciding | | τ_sel | 0.35 | abandon if > 35% of tuples survive | | τ_prog | 0.6 | only abandon while < 60% through the scan | | M_avail | 64 GB | memory budget for materialization | | τ_stop | 0.9 | stop probing a filter that rejects too little | | filter | 20 bits/key, 7 hash functions | FPR 6.1×10⁻⁵ | | probe cost | 2.48 cycles/tuple | 64-byte cache-line blocks, 32-bit sectors, VPGATHERDD | Sampling is sequential, not random — random access breaks streaming execution. On JOB, τ_sel in 0.1-0.3 gave the best speedups; τ_sel = 0 (drop almost everything) hurt. Why Anser fits — and the one place it does not | RPT+ concept | Anser today | Gap | | ----------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------- | | equivalence class of join attributes | condition_key — anser.h documents it as exactly "the optimizer-generated equivalence-class symbols" | naming only | | backward pass: build the final filter once, broadcast to every table in the class | one channel per condition, union once at the coordinator, deliver to every registered consumer | already implemented | | combining filters built by different workers | gather service folds parts by bitwise OR | already implemented (the paper is single-node; this is our addition) | | forward pass: chaining, table i reduced before it builds the filter for i+1 | channels are independent, with no ordering between them | the hard part — see below | | dynamic pipeline abandonment (Alg. 2) | never abandons | overlaps Issue 2 task 2.4 | | cascade filter, blocked/sectorized BF | classic bloom: k random bitset accesses | overlaps Issue 2 tasks 2.1/2.3 | | one filter per (table, equivalence class) | plan pass injects one producer+consumer per hash join | needs join-graph analysis in anserplan.c | Anser's backward pass is essentially free — it is what the extension already does. The forward chaining pass is where this project lives or dies, for a reason that does not exist in DuckDB: • In DuckDB, a chain hop is an in-process pipeline breaker: microseconds of coordination. • In Anser, every hop is a global barrier: all segments publish → coordinator unions → all segments receive. One hop costs a full segment→QD→segment rendezvous. JOB queries join 5-10 tables, i.e. 4-9 sequential hops. If one hop costs ~30 ms, a 7-table chain adds ~200 ms of pure latency to a query whose total runtime may be under a second. That arithmetic, with a measured hop latency, decides whether M3 below is worth attempting at all. ⚠️ Correctness rule you must not get wrong. It is tempting to do the forward chaining locally on each segment and avoid the barrier. That is only correct when the tables are co-distributed on the join key. Otherwise segment s holds only part of table _j_'s key set, and filtering table j+1 with it drops row… apache/cloudberry