#2267 [Proposal] Exploring vectorized execution for Cloudberry as a pure extension (vec_executor) New discussion created by MisterRaindrop Proposers @MisterRaindrop Proposal Status Under Discussion Abstract This is a personal exploration of one question: how far can Cloudberry get with vectorized execution if it is added only as an extension, with no change to the kernel? I built

vec_executor

, an extension that hands the slices of a query plan it can run to a vectorized engine and leaves the rest to Cloudberry's executor, so every query still runs and returns the same rows. The engine is currently based on Apache DataFusion and plugs in through existing hooks. First results are encouraging: on a single host, TPC-H at scale factor 100 runs much faster than on the same cluster without the extension, with identical results. These are basic tests only; tests with more data, more nodes and more detail are being prepared. I am posting this to share the exploration and to hear the community's views: whether the direction is worth pursuing, what concerns people most, and where the kernel could help. The code is open source under the Apache License 2.0: https://github.com/MisterRaindrop/cloudberry-datafusion Motivation Cloudberry's executor follows PostgreSQL's volcano model and processes one row at a time. For analytical queries dominated by scans, aggregates and joins, much of the CPU time goes to per-row function calls, interpreted expressions, and forming and deforming tuples. Cloudberry already has a columnar storage format, PAX, but the executor still works row by row, so the benefit of columnar storage shows up mainly in I/O. A vectorized engine processes batches of columns at a time (here in Arrow format), which removes much of the per-row overhead and can run on several threads within a segment. Rewriting the executor inside the kernel is costly and slow. This exploration asks whether Cloudberry can use a mature vectorized engine through an extension, keeping Cloudberry's semantics and operational behaviour, and if so, where the limits of that approach lie. Related work. #2065 describes pg_vexec, a vectorized executor built as an extension on a PostgreSQL 19 port of Cloudberry.

vec_executor

differs in two ways: it targets the current Cloudberry

main

as it is, without any kernel patch, and it delegates whole plan slices to a separate engine library behind a C interface rather than implementing vectorized nodes inside PostgreSQL. I would be glad to compare notes. Implementation Design principles 1. Extension only, no kernel changes. It plugs in through

planner_hook

,

ExecutorStart_hook

,

ExecutorRun_hook

and

ExplainOneQuery_hook

. Without the extension loaded, or with

vec.mode = off

(the default), Cloudberry behaves exactly as before. 2. Reuse the interconnect. Data between slices still travels through Cloudberry's Motions and interconnect; there is no separate network layer, and distributed execution keeps its semantics. When both ends of a Motion run in the engine, the Motion can carry Arrow batches instead of tuples (

vec.motion_batches

, experimental). 3. Memory and resources under Cloudberry's control. The engine's memory pool is bounded by the operator memory the coordinator assigns, and spills beyond it. Every byte the engine allocates is leased from the vmem tracker. Engine threads belong to the QE process and are subject to resource group CPU limits. Architecture The extension has two layers: • The host (

src/

, C) runs in every backend and is the only part that calls PostgreSQL. It checks whether each slice can run in the engine, translates the slices that can into a plan IR (JSON), scans tables with the query's snapshot, sends and receives Motion data on the interconnect, leases the engine's memory from the vmem tracker, and turns engine errors into `ereport`s. • An engine (

engines/<name>/

) is a separate library,

vec_engine_<name>.so

, chosen by

vec.engine

and loaded on first use. It runs a slice's plan on its own threads.

engines/datafusion

is written in Rust on Apache DataFusion;

engines/null

is written in C, runs nothing, and is used for tests. Three written contracts sit between them, so that a new engine can be built from them alone: • `include/vec_engine.h`: the function table, the rules for threads, signals, errors and memory, and the capabilities an engine declares (

VecCaps

). • `docs/ir.md`: the plan IR: nodes, expressions, types, and the PostgreSQL functions an engine must compute with PostgreSQL's semantics. • `docs/wire.md`: the Arrow IPC streams between engine slices, including cdbhash routing, on which both ends must agree byte for byte. Data passes between the host and the engine through the Arrow C Data Interface. Because the engine is replaceable, DataFusion is not a fixed choice; a C++ engine such as Velox could be added later. For PAX tables there is an optional direct reader that reads PAX files column by column on the engine's threads, without forming tuples. It is built by patching a copy of the PAX sources; neither the source tree nor the installed

pax.so

is changed. Current status What runs in the engine: • Scans of heap, AO row, AO column and PAX tables. • Joins: hash joins (inner, outer, semi, anti,

NOT IN

), which spill; merge joins, run as hash joins; nested loop joins. • Aggregates: plain, hashed and two-stage,

DISTINCT

aggregates, grouping sets, and the common aggregate functions. • Sort,

LIMIT

, window functions,

UNION ALL

/

INTERSECT

/

EXCEPT

, and correlated aggregate subqueries. • Types: integers, floats,

numeric

,

bool

, date and time types,

text

/

varchar

/

char(n)

, with PostgreSQL's semantics, including the same errors and SQLSTATEs, the same numeric scales and rounding, and short-circuit

AND

/

OR

. A slice runs in the engine only when everything in it can; otherwise the whole slice stays on PostgreSQL, and

EXPLAIN

says why. Coverage (TPC-H and TPC-DS at SF1, both planners): • All 44 TPC-H runs use the engine; in 41 of them every slice that computes anything runs in the engine. The rest are the top slices of init plans. • All 206 TPC-DS runs use the engine; in 170 of them every slice that computes anything runs in the engine. The rest are mostly top slices of init plans, and a few slices that only pass rows on and are not worth handing over. Tests: 24 regression test files, each query run with the extension on and off and the results compared; and an engine-level check that runs the same tests once with the DataFusion engine and once with the null engine and requires the same results. Size, without comments, blank lines and tests: about 14,500 lines of C in the host and about 12,400 lines of Rust in the DataFusion engine. Preliminary results These are basic tests to check whether the approach is viable, not a full performance evaluation. TPC-H SF100: one host, 3 primaries, PAX tables, compared with the same cluster without the extension. Each query ran 5 times; the table gives the sum of the 22 medians, warm cache. | Planner | Cloudberry | vec_executor | Speedup | | ---------------- | ---------- | ------------ | ------- | | Postgres planner | 1089 s | 116 s | 9.4x | | GPORCA | 1101 s | 173 s | 6.4x | • All 22 queries are faster, and every configuration returned the same output for every query. • Method and per-query figures: `docs/benchmarks.md`. • Note: the host has 20 hardware threads. With 3 primaries, Cloudberry's executor uses 3 cores, while vec_executor runs s… apache/cloudberry