Hey @Rocky We want to upgrade our starrocks cluster from 3.5.11 to 3.5.16. What considerations should we notice?

Upgrading from StarRocks 3.5.11 to 3.5.16 is a patch-version upgrade. Since both are within the v3.5 minor release, the upgrade is relatively straightforward and supports a rolling upgrade without downtime. Here are the key considerations and steps you should follow: 1. General Upgrade Order Always follow this sequence to ensure compatibility: 1. Upgrade all BEs (and CNs) first: BEs are designed to be backward compatible with FEs. Upgrade them one by one. 2. Upgrade Follower FEs: Upgrade all Follower nodes next. 3. Upgrade Leader FE: Finally, upgrade the Leader node. 2. Specific Considerations for v3.5.x * JDK 17 Requirement: StarRocks v3.5 requires JDK 17 or later. Since you are already on 3.5.11, you likely have this configured, but ensure your environment and

fe.conf

(

JAVA_OPTS

) are consistent with the new version's requirements. * Downgrade Restriction: After upgrading to 3.5.2 or later (which includes 3.5.16), do not downgrade back to 3.5.0 or 3.5.1, as this will cause metadata incompatibility and FE crashes. * Partial Updates (Primary Key Tables): Some versions in the 3.5.x cycle had adjustments to column-mode partial updates. If you use this feature heavily, verify that your data ingestion remains consistent after the upgrade. * Routine Load: v3.5.16 introduces a change where Routine Load jobs will automatically pause on non-retryable errors (like rows exceeding Primary Key size limits) instead of retrying indefinitely. 3. Pre-Upgrade Checklist Before starting the upgrade: * Backup Metadata: Back up the

meta

directory of your Leader FE. * Disable Tablet Balancing: To prevent background tasks from interfering with the rolling upgrade, run:

sql
    ADMIN SET FRONTEND CONFIG ("tablet_sched_max_scheduling_tablets" = "0");
    ADMIN SET FRONTEND CONFIG ("tablet_sched_max_balancing_tablets" = "0");

* Upgrade Test: If possible, test the upgrade on a single FE/BE in a staging environment first to verify that your specific workloads (especially External Catalogs or UDFs) behave as expected. 4. Key Improvements in 3.5.16 Moving to 3.5.16 will bring several stability fixes: * Checksum Fixes: Resolves potential checksum failures caused by DelVec CRC32 compatibility issues. * Performance: Better cloud-native tablet metadata fetch and repair efficiency. * Observability: Improved logging for

WarehouseManager

and better thread stack printing. After the upgrade is complete, remember to re-enable tablet balancing:

sql
ADMIN SET FRONTEND CONFIG ("tablet_sched_max_scheduling_tablets" = "2000"); -- Or your previous value
ADMIN SET FRONTEND CONFIG ("tablet_sched_max_balancing_tablets" = "100"); -- Or your previous value

References • release_notes/release-3.5.md

Hey @Rocky Is meta directory shared between all be pods in shared nothing arch?

In a shared-nothing architecture, the

meta

directory (and the data storage) is not shared between pods. Here is the breakdown of how storage works for FE and BE pods: 1. The

meta

Directory (FE Nodes) The

meta

directory is specific to FE (Frontend) nodes, not BE nodes. * Not Shared: Each FE pod must have its own dedicated storage (typically a unique Persistent Volume in Kubernetes). * Consistency: Although the physical files are not shared on a disk level, the FEs synchronize the metadata over the network using the BDB JE (Berkeley DB Java Edition) protocol. * Requirement: You should never point multiple FE pods to the same physical

meta

directory on a shared network drive; this will cause metadata corruption. 2. The

storage

Directory (BE Pods) BE (Backend) nodes use the

storage_root_path

(usually a

storage

directory) to store actual data tablets. * Not Shared: In a shared-nothing architecture, each BE pod manages its own local disks. * Isolation: One BE pod cannot see or access the physical data files of another BE pod. * Redundancy: Data reliability is achieved by StarRocks automatically creating multiple replicas of data and distributing them across different BE pods. Summary for Kubernetes (Pods) If you are deploying on Kubernetes: * FE Pods: Each needs a unique PVC mounted to the

meta

path. * BE Pods: Each needs a unique PVC (or multiple PVCs) mounted to the

storage

path. Why it's called "Shared-Nothing": The name comes from the fact that no two nodes (FE or BE) share the same physical storage or memory. They only communicate over the network. This eliminates the "single point of failure" or "performance bottleneck" of a central shared filesystem. *Note: If you were using the Shared-Data (Cloud-Native) architecture, the BEs (called CN nodes there) would share a central storage like S3 or HDFS for data, but the FE metadata would still remain local to each FE node.* References • administration/management/BE_configuration.md • faq/Deploy_faq.md • faq/Others.md • faq/operation_maintenance_faq.md

@Rocky Then What is meta directory in be pods?

In a StarRocks Backend (BE) pod, the

meta

directory (found inside your

storage_root_path

) is used to store Local Tablet Metadata. While the FE

meta

directory stores the global state of the cluster (database/table schemas, user permissions, etc.), the BE

meta

directory stores the specific technical details of the data chunks (tablets) that live on that specific node. 1. What is inside the BE

meta

directory? It contains a RocksDB database. This local database tracks: * Tablet Information: Which tablets are stored on this disk. * Rowset Metadata: Which data files (segments) belong to which version of a tablet. * Version History: The consistent versions available for queries. * Schema Hash: Local validation to ensure data matches the expected table structure. 2. Why does each BE need its own? In a Shared-Nothing architecture, each BE is responsible for its own physical disks. * When a BE starts, it reads this

meta

directory to "know" what data it has. * Without this directory, the BE would just see a bunch of

.dat

files in the

data/

folder and wouldn't know which table they belong to or which version of the data is current. 3. Comparison: FE Meta vs. BE Meta Feature | FE

meta

Directory | BE

meta

Directory Scope | Global: Entire cluster metadata. | Local: Only tablets on this node. Content | BDB JE (Berkeley DB) logs and images. | RocksDB (Tablet & Rowset metadata). Consistency | Replicated across Follower FEs. | Local to the node (replicated via StarRocks data replicas). Criticality | Losing this loses your table definitions. | Losing this loses the data on this specific node. Summary for your Upgrade During your upgrade from 3.5.11 to 3.5.16: * Do not move or share the BE

meta

directory between pods. Each BE pod must have its own persistent volume for its

storage_root_path

. * Cleanup: You should never manually delete the

meta

directory in a BE unless you intend to wipe the data on that node and let the cluster repair it from other replicas. * Cloud-Native Note: If you were using Shared-Data mode, you would also see a

meta/

folder in your S3/Object Storage bucket, which serves a similar purpose but is accessible by all compute nodes. References • administration/management/BE_configuration.md • introduction/Architecture.md • administration/management/FE_configuration.md References • administration/management/BE_configuration.md • faq/Deploy_faq.md • faq/operation_maintenance_faq.md • faq/Others.md