@Rocky How does using tables with shared-data mode (S3 Storage) vs using local disks in shared nothing mode compare? What are the tradeoffs and things I should be aware of?

In StarRocks, the choice between Shared-nothing (classic architecture with local disks) and Shared-data (compute-storage separation with S3/Object Storage) is primarily a tradeoff between performance predictability and operational elasticity/cost. Here is a detailed comparison to help you decide: 1. Architectural Comparison Feature | Shared-nothing (Classic) | Shared-data (Compute-Storage Separation) Storage | Local disks (SSD/HDD) on BE nodes. | Object Storage (S3, OSS, GCS, Azure Blob) or HDFS. Node Type | BE (Backend): Handles both storage and compute. | CN (Compute Node): Stateless; handles compute and local caching. Scaling | Scaling storage requires adding BEs and rebalancing data. | Scaling compute (CNs) is **instant**; no data migration is needed. Reliability | Depends on StarRocks replication (e.g., 3 replicas). | Leverages 99.999999999% durability of S3/Object storage. ────────── 2. Performance Tradeoffs * Shared-nothing: Provides the most consistent high performance because data is always local. It is ideal for ultra-low latency requirements where you want to avoid any network overhead for storage I/O. * Shared-data: Performance depends heavily on the Data Cache. * Cache Hit: Performance is comparable to shared-nothing (often within 10-20% or even faster in some optimized scenarios). * Cache Miss: Performance is limited by network bandwidth and object storage latency. * Feature Note: StarRocks v3.3+ includes "Data Cache Warmup," allowing you to proactively pull data into the local cache (using

CACHE SELECT

) before running queries. 3. Key Tradeoffs & Considerations Operational Elasticity (The "Shared-data" Advantage) In shared-data mode, because CN nodes are stateless, you can use Auto-scaling or Spot Instances for your compute layer. If a CN node fails, you don't lose data and don't need to wait for a replica to rebuild; a new CN simply starts up and begins fetching data from S3. Cost Efficiency * Storage Cost: S3 is significantly cheaper than high-performance local NVMe/SSDs. * Compute Cost: Shared-data allows you to shut down compute nodes during idle hours without losing your data, which is not possible in shared-nothing mode. Primary Key Tables Shared-data mode now supports Primary Key tables (starting from v3.1), including partial updates. However, because the "state" (the primary key index) needs to be managed, StarRocks uses Persistent Index (on S3/Local Cache) to ensure performance doesn't degrade. Maintenance & Limitations * No Rebalancing: In shared-nothing, adding a node triggers a "Clone" process to move data tablets. In shared-data, adding a node is nearly instantaneous. * Backup/Restore: Currently, shared-data clusters have different support levels for native `BACKUP`/`RESTORE` commands compared to shared-nothing. You should rely on the versioning and lifecycle policies of your object storage for disaster recovery. * Data Cache Management: You must manage the local disk space on CN nodes for the cache. If the disk is too small, you'll experience frequent cache evictions and "cold" query performance. 4. When to Choose Which? Choose Shared-nothing (Local Disks) if: * You are deploying on-premises with high-performance local hardware. * You have extreme, sub-second latency requirements for every single query. * Your storage-to-compute ratio is stable and doesn't change much. Choose Shared-data (S3 Storage) if: * You are deploying in the Cloud (AWS, GCP, Azure). * Your data volume is growing much faster than your compute needs. * You want to save costs by using Spot instances or scaling down compute at night. * You want to avoid the operational headache of data rebalancing during cluster expansion.

@Rocky Does StarRocks write to disk or the volume I made for storage if I have

storage_root_path

set. I see its being used for data caching not sure what else its used for in shared data mode

In shared-data mode (Cloud-native StarRocks), the usage of the disk volume pointed to by

storage_root_path

changes significantly compared to the traditional shared-nothing mode. Here is exactly what StarRocks uses that volume for in shared-data mode: 1. Data Caching (Primary Use) As you've observed, the primary resident on this disk is the Data Cache (also called Block Cache or Starlet Cache). * Path: Usually stored under

<storage_root_path>/starlet_cache/

or

<storage_root_path>/datacache/

. * Purpose: To avoid fetching data from object storage (S3/GCS/HDFS) for every query, StarRocks caches frequently accessed data blocks locally. * Persistent Data: The actual "source of truth" for your data is written to your object storage, not this local disk. If you delete the local cache, StarRocks will simply re-fetch the data from the cloud when needed. 2. Local Metadata and System State Even though data is remote, the BE/CN node still needs to track local state. * Path:

<storage_root_path>/meta/

* Purpose: It stores lightweight metadata about the tablets the node is currently responsible for, versioning info, and internal BE logs/traces. This is usually very small (MBs to a few GBs) compared to the actual data. 3. Intermediate Query Results (Spill-to-Disk) If you enable the Spill to Disk feature (for large joins or sorts that don't fit in memory), StarRocks will use local disk space for temporary storage. * Configuration: This is controlled by

spill_local_storage_dir

in

be.conf

or

cn.conf

. * Behavior: If you haven't explicitly set a separate path for spilling, it may default to directories on the same volume or within your StarRocks installation path. 4. Temporary Scratch Space The parameter

query_scratch_dirs

(introduced in v3.2.0) also defines where temporary files are written during query execution. Like the spill directory, this is for transient data that is deleted once a query completes. Summary Checklist * Does it write persistent data to

storage_root_path

? No. All persistent table data goes to your Object Storage Volume. * Is it safe to use a smaller/faster disk? Yes, because it's just a cache. However, the size of this disk directly impacts query performance—a larger cache means fewer remote fetches. * What happens if the volume is lost? You won't lose data. The BE will restart, find its cache empty, and begin populating it again from the object store as queries run. Rocky's Tip: In shared-data mode, you should treat the

storage_root_path

volume as ephemeral but high-performance storage (like an NVMe SSD) to get the best out of the Data Cache! References • administration/management/resource_management/spill_to_disk.md • administration/management/BE_configuration.md • knowledge/usage/best-practice.md • sql-reference/System_variable.md

@Rocky I am backfilling a table in shared data mode the same way I was in shared nothing mode and the data is being loaded but the publish step keeps failing on this one specific table with

ErrMsg: Fail to publish partition 26232 error Fail to publish version for tablets:[], error msg: link rpc channel failed, host: starrocks-shared-data-cluster-cn-2.starrocks-shared-data-cluster-cn-search.starrocks.svc.cluster.local

It worked for all the other tables. The table this is failing on is quite large though. The compute nodes are fine too and have not crashed. What could be going on.

The error message

link rpc channel failed

followed by a Kubernetes internal DNS hostname suggests that the FE (Frontend) is failing to establish a network connection to the CN (Compute Node) during the Publish Version phase. Since this only happens on your largest table, it is likely that the "Publish" request for this table is hitting a limit or a timeout due to the sheer volume of tablet metadata being processed or the concurrency of requests hitting the CN. Here are the most likely causes and solutions: 1. FE RPC Connection Pool Exhaustion When backfilling a large table, the FE might be trying to open many concurrent RPC connections to the same CN. In StarRocks, there is a limit on the number of pooled

brpc

connections. * Action: Increase the connection pool size in your

fe.conf

. * Parameter:

brpc_connection_pool_size

* Suggested Value: Increase from the default

16

to

64

or

128

. * Note: This requires an FE restart. 2. Large Metadata Body Size If a single partition in your large table has an extremely high number of tablets, the RPC payload (the list of tablets and their metadata) sent from FE to CN might be very large. * Action: Check the

brpc_max_body_size

on your CNs (`be.conf`/`cn.conf`). * Parameter:

brpc_max_body_size

* Default:

2147483648

(2GB). It is unlikely you are hitting 2GB for one partition, but if your metadata is unusually dense, check the CN logs (

<http://be.INFO|be.INFO>

) for any messages about "body size too large". 3. RPC Timeouts under High Load The CN might be under high CPU/IO load due to the backfilling of the large table, causing it to be slow in responding to the RPC handshake from the FE. * Action: Increase the tablet writer RPC timeout on the CNs. * Parameter:

tablet_writer_rpc_timeout_sec

* Suggested Value: Increase it (e.g., to

600

or higher). 4. Shared-Data Batch Publish (v3.2+) In shared-data mode, StarRocks attempts to batch several "publish" tasks together to improve throughput. If a batch becomes too large or complex for a massive table, it might trigger failures. * Action: Try disabling batch publishing to see if it stabilizes the backfill. * Parameter (FE):

lake_enable_batch_publish_version = false

* Mutation: This is a runtime mutable configuration. You can set it via:

sql
    ADMIN SET FRONTEND CONFIG ("lake_enable_batch_publish_version" = "false");

5. Kubernetes DNS/Networking Latency The error

link rpc channel failed

to a

.local

address sometimes indicates a DNS resolution failure or an intermittent network drop within the K8s overlay network. * Check: Ensure your

priority_networks

is configured correctly in both FE and BE/CN configs to use the correct CIDR for your K8s pod network. This ensures StarRocks doesn't try to use a "wrong" IP that might be intermittently reachable. * Check: Check if the CN pod

starrocks-shared-data-cluster-cn-2

is experiencing high memory usage or OOM kills (even if it didn't stay down, a restart would drop existing RPC links). Summary of Next Steps: 1. Check the FE log (

fe.log

) for more details around that timestamp. It might show if the RPC failed due to a timeout or a rejected connection. 2. Check the CN log (

<http://be.INFO|be.INFO>

) on

starrocks-shared-data-cluster-cn-2

to see if it was experiencing heavy GC or "server overcrowded" errors. 3. Try setting

lake_enable_batch_publish_version = false

on the FE to see if the partition finally publishes. References • administration/management/FE_configuration.md

@Rocky How do I check if I need to increase my "write_buffer_size" or "tablet_writer_rpc_timeout_sec" size

To determine if you need to adjust these parameters, you should monitor your data loading logs and cluster health for specific symptoms. Both

write_buffer_size

and

tablet_writer_rpc_timeout_sec

are critical for balancing ingestion throughput and system stability. 1. When to increase

write_buffer_size

This parameter (default: 100MB) determines the threshold at which the in-memory buffer (MemTable) is flushed to disk as a segment file. * Increase it if: * High File Count: You notice that each load job generates a large number of small files (segments) on the BE. This can lead to metadata pressure and slow query performance (as StarRocks has to scan many small files). * IOPS Bottlenecks: Your disk IOPS is hitting limits during ingestion because the system is performing too many small, frequent writes. * Decrease it if: * Memory Pressure: You are encountering "Memory Limit Exceeded" errors during heavy ingestion. * RPC Timeouts: If this value is set too high (e.g., several GBs), the batch of data sent via RPC becomes so large that it may exceed the default RPC timeout. 2. When to increase

tablet_writer_rpc_timeout_sec

This parameter (default: 600s) controls the timeout for the RPC that sends data batches to the BEs. * Increase it if: * Timeout Errors: Your load jobs fail with error messages like

RPC timeout

,

failed to send batch

, or

tablet writer add batch timeout

. * Large Data Batches: You have already increased

write_buffer_size

or

max_tablet_write_chunk_bytes

to improve throughput, which naturally makes the data transfers take longer. * Network/Disk Latency: You are running on a cluster with high network latency or slow disks where writing the buffer to the remote BE consistently exceeds the default time. Summary of Symptoms and Actions Symptom | Probable Cause | Action

RPC timeout

or

failed to send batch

errors | Data transfer takes longer than the timeout | Increase

tablet_writer_rpc_timeout_sec

Performance degradation / Too many small files |

write_buffer_size

is too small, causing frequent flushes | Increase

write_buffer_size

TabletWriter add batch with unknown id

| Writer process timed out waiting for data | Increase

streaming_load_rpc_max_alive_time_sec

How to check current values: You can check these settings in your

be.conf

file or by running the following SQL command:

sql
SHOW BACKENDS\G
-- Look for the Config column or check via the BE debug port (default 8040)

Recommendation: If you increase

write_buffer_size

significantly to improve throughput for wide tables, it is generally recommended to increase

tablet_writer_rpc_timeout_sec

concurrently to ensure the larger data batches have enough time to be processed. References • administration/management/BE_configuration.md • faq/loading/Loading_faq.md

@Bailey Kocin one more thing to add is that we are not currently investing anymore in new features for shared-nothing and shared-data with the disk cache it is almost as performant. The few gaps in performance in shared-data will be addressed in 4.1. Also we have a new feature called cache replica which is going to solve for when you lose a node and need to replace it, there will be cache replicas so you don't go back to a cold cache. These are all things Rocky doesn't know so I wanted to complement the answer.