@Rocky Would it be okay to deploy StarRocks on Rocky Linux 9 or Ubuntu 26.04 without using Docker?

Yes, you can deploy StarRocks on Rocky Linux 9 or Ubuntu 26.04 without using Docker. StarRocks is designed to run natively on Linux as a set of binary processes. Rocky Linux 9 Rocky Linux 9 is a binary-compatible downstream of Red Hat Enterprise Linux (RHEL) 9. While some older documentation may specifically list RHEL/CentOS 7.9, StarRocks is widely compatible with modern RHEL-based distributions like Rocky Linux 8 and 9. * Recommendation: Rocky Linux 9 is a stable and reliable choice for production deployments. Ubuntu 26.04 Ubuntu 26.04 (released in April 2026) is the latest LTS version. StarRocks binaries are typically built with compatibility for modern Linux kernels and glibc versions found in current Ubuntu releases. * Note: While the official documentation might still mention Ubuntu 22.04 or 24.04 as the "supported" versions, StarRocks' core dependencies (C++ for the BE and Java for the FE) are standard and typically work seamlessly on newer LTS releases. Key Prerequisites for Non-Docker Deployment To ensure a successful manual deployment, make sure your environment meets these requirements: 1. CPU (AVX2): The Backend (BE) process relies on the AVX2 instruction set for its vectorized execution engine. Check your CPU support with:

bash
    cat /proc/cpuinfo | grep avx2

2. Java (JDK): The Frontend (FE) requires a Java runtime. * For StarRocks v3.5 and later (including v4.x), you must use JDK 17 or later. * StarRocks does not support JRE; the full JDK is required. 3. System Limits (ulimit): You must increase the maximum file descriptors and processes. Typical production settings include:

bash
    # /etc/security/limits.conf
    * soft nofile 655350
    * hard nofile 655350
    * soft nproc 65535
    * hard nproc 65535

4. Architecture: If you are using ARM64 (e.g., AWS Graviton), you may need to compile the binaries yourself or use specific ARM builds, as the standard release tarballs are often optimized for x86_64. Deployment Method You can download the official tarballs from the StarRocks Download page and follow the Manual Deployment guides in the documentation for either Shared-nothing or Shared-data architectures. References • deployment/deployment_prerequisites.md • developers/build-starrocks/build_starrocks_on_ubuntu.md • developers/build-starrocks/Build_in_docker.md • deployment/environment_configurations.md • faq/Deploy_faq.md

@Rocky When deploying on bare metal, is using Docker the recommended approach?

For bare metal production deployments, using standalone Docker is generally not the primary recommended approach. Instead, the two standard paths for production are: 1. Manual (Native Binary) Deployment – Recommended for Bare Metal Deploying the binaries directly onto the host OS is the most common and often preferred method for bare metal environments. * Performance: It provides direct access to hardware without the overhead of Docker’s networking or storage drivers. Since StarRocks is a high-performance OLAP engine, avoiding these abstractions helps ensure the lowest possible latency. * System Control: StarRocks requires specific OS-level tuning (e.g., disabling Transparent Huge Pages, setting

ulimits

, and configuring

sysctl

for memory overcommit). These are more straightforward to manage and verify on a native host than inside a container. * Standard Practice: The official production deployment guide lists "Manual Deployment" as the primary option for shared-nothing clusters on physical hardware. 2. Kubernetes (Operator) – Recommended for Orchestration If your organization requires containerization for production, the recommended approach is to use the StarRocks Operator on Kubernetes rather than standalone Docker. * Automation: The Operator handles complex lifecycle tasks like rolling upgrades, scaling, and health monitoring, which you would otherwise have to manage manually with standalone Docker. * Best of Both Worlds: When configured correctly (e.g., using

hostNetwork: true

and local PVs), Kubernetes can provide near-bare-metal performance with the operational benefits of containers. When should you use standalone Docker? Standalone Docker (via

docker run

or

docker-compose

) is excellent for: * Development & Testing: Quickly spinning up an environment on a laptop. * Proof of Concept (PoC): Evaluating StarRocks features without needing a full cluster setup. * Non-Performance-Critical Tasks: Small-scale environments where ease of deployment is more important than absolute performance. Summary: For a high-performance production cluster on Rocky Linux 9 or Ubuntu 26.04, you should deploy manually using the native binaries unless you have an existing Kubernetes infrastructure. References • faq/Deploy_faq.md • quick_start/shared-nothing.md • deployment/deployment_overview.md • developers/development-environment/dev-env-setup.md • deployment/environment_configurations.md

@Rocky When deploying BE, is numactl-based optimization recommended?

While the official StarRocks documentation does not mandate

numactl

, using it is highly recommended for bare metal deployments on multi-socket servers. Because StarRocks is currently not NUMA-aware, it does not internally optimize for memory locality across different CPU sockets. This can lead to two main performance issues: 1. Remote Memory Latency: A thread running on CPU socket 0 might frequently access memory residing on socket 1, which is significantly slower than local access. 2. Memory Imbalance: The OS might allocate memory primarily from one NUMA node until it is full, triggering aggressive reclamation or swapping while other nodes have plenty of free memory. Recommended Approaches Depending on your hardware and operational complexity, there are two common ways to use

numactl

with StarRocks Backend (BE): 1. The "Interleave" Strategy (Easiest) This is the most common recommendation for systems that are not NUMA-aware. It instructs the OS to spread memory allocations evenly across all available NUMA nodes. * How to run:

bash
    numactl --interleave=all ./bin/starrocks_be --daemon

* Benefit: Prevents a single NUMA node from becoming a bottleneck or running out of memory prematurely. It provides a more predictable, though not perfectly local, performance profile. 2. The "Multi-Instance" Strategy (Best Performance) If you have a very large machine (e.g., 2 sockets with 64 cores each), you can achieve better performance by running one BE instance per NUMA node. * How to run: You would configure two separate BE directories and bind each to a socket and its local memory:

bash
    # Instance 1 pinned to Node 0
    numactl --cpunodebind=0 --membind=0 ./be_1/bin/starrocks_be --daemon

    # Instance 2 pinned to Node 1
    numactl --cpunodebind=1 --membind=1 ./be_2/bin/starrocks_be --daemon

* Benefit: Ensures 100% memory locality for each process. * Trade-off: Increases management complexity (you must manage multiple BE ports and configurations) and divides the BE's internal caches (like PageCache). Summary Recommendation * For most users: Use

numactl --interleave=all

. It provides a significant safety net against memory imbalances with zero configuration changes. * For performance enthusiasts: If your benchmarks show cross-socket latency is a bottleneck, consider the multi-instance approach. Note: Ensure

numactl

is installed on your Rocky Linux or Ubuntu host (

sudo yum install numactl

or

sudo apt install numactl

). References • administration/management/BE_configuration.md • deployment/plan_cluster.md • faq/Others.md • administration/management/resource_management/Memory_management.md • deployment/deployment_overview.md References • deployment/plan_cluster.md • administration/management/resource_management/Memory_management.md • administration/management/BE_configuration.md

@Rocky In the multi-instance approach, what does it mean that the BE internal cache needs to be partitioned? Could you explain in detail?

In the multi-instance approach (running multiple BE processes on a single physical machine), "partitioning the internal cache" refers to the fact that each BE process operates as an independent entity with its own isolated memory space. When you split one large BE into two or more smaller BEs, the system's global resources (RAM) are divided, and so are the caches. Here is a detailed breakdown of what that means: 1. Isolated LRU Caches StarRocks BEs maintain several internal caches to speed up performance. In a multi-instance setup, these are not shared between processes: * Page Cache: Stores recently read data pages from disk. * Query Cache: Stores results of sub-queries or query fragments. * Metadata Cache: Stores schema information and tablet metadata. The Impact: If BE-1 and BE-2 are on the same machine, BE-1 cannot serve a request using data already cached in BE-2's memory. Even though the data is physically on the same machine, the processes cannot "see" each other's RAM. 2. Reduced "Global" Cache Efficiency In a single-instance setup, one large Page Cache (e.g., 256GB) can dynamically allocate space to whatever data is "hottest" across the entire machine. In a multi-instance setup: * You must manually set memory limits for each instance (e.g., two BEs with 128GB each). * If BE-1 is under heavy load and needs more cache, it cannot borrow unused memory from BE-2's cache. * This can lead to a "fragmentation" of your hot data; BE-1 might be evicting useful pages because its smaller 128GB limit is full, while BE-2 has plenty of free space. 3. Data Distribution (Tablet Mapping) The StarRocks Frontend (FE) treats each BE process as a separate node. * Data Partitioning: The FE will distribute different tablets (data shards) to each BE. This means BE-1 will naturally cache the data for "Tablets A, B, C" and BE-2 will cache "Tablets D, E, F." * No Duplication (Mostly): Because each BE typically manages different data, you won't usually waste memory by having the exact same data cached twice on the same machine. * However, if you have high-replication factors (e.g., 3 replicas), the FE might accidentally place two replicas of the same data on two different BEs on the same machine. This would be a waste of both disk and cache. 4. Configuration Requirements To prevent the instances from crashing or competing for the same RAM, you must explicitly "partition" the configuration in each `be.conf`: Parameter | Single Instance | Multi-Instance (Example for 2 BEs)

mem_limit

| 90% (Default) | 45% per instance

storage_page_cache_limit

| 20% (Default) | 10% per instance

be_port

| 9060 | BE1: 9060, BE2: 19060

storage_root_path

|

/data/storage

| BE1:

/data1

, BE2:

/data2

Summary: Is it worth it? * The Benefit: You get better NUMA locality. Threads on Socket 0 only access memory on Socket 0, which is much faster. * The Cost: You lose cache flexibility. You have to manage more configuration files and potentially deal with slightly lower "hit rates" because the cache is split into smaller, rigid pools. If your machine has more than 64 cores or multiple physical CPU sockets, the performance gain from NUMA locality usually outweighs the slight loss in cache efficiency. For smaller machines, a single instance is almost always better. References • deployment/deploy_manually.md • faq/Others.md • faq/Deploy_faq.md • deployment/plan_cluster.md • deployment/upgrade.md • deployment/downgrade.md

@Rocky Is there a way to throttle resource usage (especially network) during backup and restore operations?

StarRocks does not currently provide a direct "bandwidth cap" (e.g., a specific KB/s limit) for

BACKUP

and

RESTORE

operations. Instead, throttling is achieved by controlling the concurrency of tasks and threads. By reducing the number of parallel workers, you effectively reduce the network, disk I/O, and CPU pressure these operations exert on the cluster. 1. Throttling via BE Configurations You can limit the number of threads performing the actual data transfer (upload/download) on each Backend node in `be.conf`: * For BACKUP:

upload_worker_count

* Default:

0

(automatically set to the number of CPU cores). * To Throttle: Set this to a small fixed value (e.g.,

1

or

2

) to limit how many files a BE uploads to remote storage (S3/HDFS) simultaneously. * For RESTORE:

download_worker_count

* Default:

0

(automatically set to the number of CPU cores). * To Throttle: Set this to a small value to limit concurrent downloads from the repository. * Snapshot Concurrency:

make_snapshot_worker_count

(Default:

5

) controls the threads creating the local data snapshots before the upload starts. 2. Throttling via FE Configurations You can also limit how many total tasks the Frontend (FE) assigns to each BE for a single job by modifying these dynamic FE parameters (using

ADMIN SET FRONTEND CONFIG

): *

max_upload_task_per_be

: The maximum number of upload tasks assigned to a BE node in a single

BACKUP

operation. *

max_download_task_per_be

: The maximum number of download tasks assigned to a BE node in a single

RESTORE

operation. Recommended Strategy Because there is no "hard speed limit," the best practice is to: 1. Schedule during off-peak hours: StarRocks documentation explicitly recommends running these jobs when the cluster is not heavily loaded. 2. Start with low concurrency: If you are worried about network saturation, set

upload_worker_count

and

download_worker_count

to

1

or

2

and monitor the impact before increasing. 3. Monitor Network I/O: Use system tools (like

nload

or

iftop

) to observe the actual throughput when these worker counts are adjusted. References • administration/management/BE_configuration.md • administration/management/Backup_and_restore.md • administration/management/FE_configuration.md • faq/operation_maintenance_faq.md References • administration/management/BE_configuration.md • administration/management/Backup_and_restore.md • administration/management/FE_configuration.md