Hi All! Last months I spent to platformize Cloudberry — simplifying deployment and operation tasks and extending it with HashiCorp Vault. So I built a Kubernetes operator that turns all of that into a single declarative

CloudberryCluster

resource. Currently it is pre-release and i still working on data-loading, monitoring , helm tuning and bugs fixing. https://github.com/vyrodovalexey/cloudberry-k8s What you get: • Deploy — Helm install + one manifest, no manual

gpinitsystem

• Scale — change a field: scale-out w/ data redistribution, scale-in w/ PVC handling • Upgrade — rolling (mirrors → primaries → standby → coordinator) with auto-rollback • HA — segment mirroring, FTS auto-failover, WAL-streaming standby, segment recovery • Backup/restore — to S3 or PVC via native `gpbackup`/`gprestore`; on-demand, scheduled, retention • CLI —

cloudberry-ctl

instead of SSH-ing into pods Security — Vault-first 🔒 Cluster TLS issued by Vault PKI with auto-rotation, backup creds pulled from Vault (placeholder-only on disk, resolved at runtime), Vault Kubernetes auth — no bootstrap tokens. Auth — Keycloak Dual-mode Basic + OIDC: JWT validation w/ JWKS caching + role-claim extraction, mapped to a five-tier permission model. Observability — out of the box 📊 Prometheus metrics (reconciliation, FTS, scaling, mirroring, certs, backups); Cloudberry-tailored

postgres-exporter

+

cloudberry-query-exporter

sidecars; metrics → VictoriaMetrics (vmagent), logs → VictoriaLogs (Vector); Some raw Grafana dashboards. OpenTelemetry OTLP tracing (gRPC/HTTP) with span error recording via an OTel Collector. Batteries included — prepared images Ready-to-run images: the original Cloudberry DB built from the official Apache Cloudberry RPM on Rocky Linux (with

zstd

for backups), a backup-toolchain image (`gpbackup`/`gprestore`/`gpbackup_s3_plugin`/`gpbackman`), plus operator + CLI images. Happy to take questions or feedback 🙏

This is genuinely awesome — huge congrats! 🎉 Pulling FTS failover, scale-out with redistribution, and ordered rolling upgrades into one CloudberryCluster resource is no small feat — that’s a way harder problem than the usual single-node Postgres operator, and it’s clear you really get the Cloudberry internals. Love the Vault-first touch (placeholder-only creds on disk, resolved at runtime) and that you leaned on native gpbackup/gprestore instead of reinventing them. Really nice work. 👏 A few things we got curious about while reading (we hack on the engine side, so this stuff is right up our alley 😄): 1. Scale-out redistribution — does it ride on gpexpand under the hood, or is it your own path? And is it online, or does it grab the redistribution lock per table? Also wondering what happens if the operator gets killed mid-redistribution — does it pick back up cleanly? 2. FTS + K8s probes — how do you keep FTS and Kubernetes liveness probes from stepping on each other? Curious whether the operator treats FTS as the source of truth for segment state, so a pod restart can’t race with an FTS failover. No rush at all — would love to see it once data-loading and the dashboards land.

Thank you! Very good questions!!! 1. Scale-out redistribution. The operator doesn't shell out to

gpexpand

despite a code comment calling it "the gpexpand equivalent." Instead it runs a controller-driven state machine (scaling-sts → registering → redistributing → completed) that ends in SQL-based redistribution. Once new pods come up and segments are registered into

gp_segment_configuration

,

RedistributeData

walks every user database and issues per-table `ALTER TABLE … SET DISTRIBUTED BY/RANDOMLY`; scale-in adds a step that lowers

numsegments

and forces a

REORGANIZE

to push data off high-numbered segments. It is explicitly not fully online. Each

ALTER TABLE … SET DISTRIBUTED

takes an

ACCESS EXCLUSIVE

lock on that table while it's rewritten (standard Cloudberry/Greenplum behavior, not relaxed by the operator). The cluster stays up and tables are processed one at a time, with a

Parallelism

knob for cross-table concurrency, but any individual table is locked during its rewrite. Progress is tracked via the

cloudberry_redistribution_progress

gauge; there's no per-table online/concurrent-DML mode. Resume after an operator crash works, but at phase granularity. Scale state lives in the

<http://avsoft.io/scale-state|avsoft.io/scale-state>

CR annotation rather than operator memory, so on restart

checkScaleProgress

re-reads it and re-enters at the recorded phase. A crash during

redistributing

re-runs

RedistributeData

, which is idempotent (re-applying an existing distribution policy is effectively a no-op, and per-table failures are logged-and-skipped). A

scaleTimeout

guard prevents indefinite hangs. Caveat: resume is phase-level, not table-level, so a mid-redistribution crash re-scans all tables and re-locks the interrupted one. 2. FTS + Kubernetes probes. FTS is the single source of truth for segment state; K8s probes are deliberately decoupled and conservative so the two don't collide. The liveness probe is intentionally minimal — a

TCPSocket

check on the Postgres port (InitialDelay 30 / Period 10 / Timeout 5, FailureThreshold 3). It only asks whether the process accepts TCP connections, not whether the segment is healthy or primary. A segment that FTS has already failed over still answers TCP, so liveness won't restart it — this is the main anti-collision mechanism. All up/down/role decisions come exclusively from

gp_segment_configuration

(read via

runFTSProbe → probeSegmentConfigWithRetries → analyzeSegments

), driven by Cloudberry's internal FTS scan and then verified against the catalog before updating status/metrics. Probe cadence is FTS-configurable (default 60/20/5) and independent of kubelet timings, so the 5-attempt retry budget tolerates transient blips without flapping. Can a pod restart race an FTS failover? Mostly no, with one honest caveat. Because liveness is TCP-only and FTS owns role state, kubelet restarts and FTS failover act on different signals and don't fight over declaring a segment down; FTS observes any restart via the catalog on its next scan. The caveat is that the operator doesn't currently gate kubelet restarts on FTS state — there's no startup-probe coordination or "don't restart during in-flight failover" interlock. So a genuinely crash-looping primary can restart while an FTS scan is in progress. The harmful race (a probe killing a healthy-but-failed-over segment) is made impossible by design; the remaining window — a crashing pod restarting mid-scan — resolves by convergence (catalog-as-truth, with failover state tracked in

<http://avsoft.io/failover-state|avsoft.io/failover-state>

) rather than strict serialization.