@Rocky I'm testing a migration of my primary key table from shared nothing to shared data. I have provisioned enough EC2 i4i.8xlarge machines such that the total NVMe capacity will always be enough to house all of my data TTL, never need to read from S3. However, very high throughput ingestion (inserts and partial updates) still go through the CN nodes. Since everything will always live on the instance store, could my partial updates theoretically perform better on the shared data architecture? Or am I still CPU bound for this task? I'm wondering how much does the NVMe I/O improvement over shared nothing EBS improvement help partial update resource usage

In short, yes, your partial updates will theoretically perform better on the Shared Data (SD) architecture with NVMe instance stores compared to a Shared Nothing (SN) architecture using EBS—provided your bottleneck was previously I/O latency or throughput. However, once you eliminate the I/O bottleneck with NVMe, you will almost certainly become CPU-bound for this specific task. Why Shared Data + NVMe Helps Partial Updates Partial updates on Primary Key tables are I/O-intensive because of the "Read-before-Write" nature of Row Mode or the high compaction overhead: 1. Lower Read Latency for Merges: In Row Mode, StarRocks must read the existing row to merge it with the updated columns. On SN with EBS, this "read" is subject to EBS latency. On SD with an NVMe cache, this read hits the instance store directly. Since your TTL ensures data is always in the cache, you're effectively operating at the speed of local NVMe for the most critical part of the update cycle. 2. Compaction Efficiency: High-throughput ingestion creates many small versions. The background compaction process, which is critical for keeping Primary Key tables healthy, involves heavy reading and rewriting. Moving this workload from EBS to NVMe (via the SD cache) significantly reduces the time threads spend in

iowait

, allowing them to cycle back to ingestion tasks faster. 3. Persistence of PK Index: Since StarRocks v3.2, Shared Data supports Persistent Indexes on local disks. On an

i4i

instance, these index lookups will be extremely fast compared to EBS-backed persistent indexes, further reducing the update latency. The Shift to CPU Bottlenecks Once the NVMe I/O removes the "waiting for disk" factor, the following CPU-intensive operations will become your primary limiters: * Primary Key Index Lookups: Even with a fast disk, calculating hashes and searching the index for every incoming row is a high-CPU task. * Delete Vector (DelVector) Generation: Calculating which rows to mark as deleted for every batch requires significant computation. * Data Compression/Decompression: As rows are read from the cache and merged into new segments, the CPU must handle Zstd/LZ4 decompression and re-compression. Architecture Optimization Tip: Column Mode If you are on StarRocks v3.3.1 or later, Shared Data supports Column Mode for partial updates. * Row Mode (Default): Reads the entire row, merges, and writes a new one. (Better for many columns, few rows). * Column Mode: Only writes the changed columns as new files and merges them during read/compaction. (Better for few columns, many rows). If your high-throughput updates only touch a few columns, switching to Column Mode in your Shared Data cluster can drastically reduce the amount of data being "processed" (CPU) and "cached" (I/O), potentially providing the largest performance boost. Summary By using

i4i

instance stores for the Shared Data cache, you've essentially given StarRocks the fastest possible "local disk" without the data loss risks of Shared Nothing on ephemeral storage (since S3 is your source of truth). You will likely see lower commit latencies and higher ingestion throughput, but you should monitor your CPU utilization closely, as it will be the next resource to saturate. References • deployment/shared_data/feature-support-shared-data.md • loading/Load_to_Primary_Key_tables.md • sql-reference/sql-statements/table_bucket_part_index/UPDATE.md • table_design/table_types/primary_key_table.md • best_practices/primarykey_table.md

@Rocky I've tested column mode before with partial updates. It performed way worse since I verified it was incompatible with merge commit. I tested stream loads with and without merge commits and column mode only worked without it. Furthermore, when I disabled merge commit on column mode, compaction scores spiked to 2000 and above. The column partial update duration was climbing to 40-50s, and likely blocking compaction work, causing a ton of rowsets to build up before the CPUs had a chance to perform compaction

Your experience highlights a very specific architectural trade-off in StarRocks. Since you have verified that Column Mode is currently incompatible with Merge Commit in your environment (which is a known limitation in several 3.x versions, specifically when combined with features like

merge_condition

), you are stuck choosing between two different bottlenecks. Here is the technical breakdown of why your partial updates will likely perform better on Shared Data + NVMe even if you remain in Row Mode, and why Column Mode failed you. 1. Row Mode: From I/O-Bound to CPU-Bound In a Shared Nothing (SN) architecture with EBS, Row Mode partial updates are almost always I/O-latency bound. Every update triggers a "Read-before-Write": the system must fetch the existing row from disk, merge it in memory, and write it back. EBS latency (typically 1–10ms) makes this slow at high throughput. By moving to Shared Data (SD) with NVMe instance stores: * Zero-Latency Reads: Because your TTL fits entirely in the NVMe cache, the "Read" part of the cycle hits the local instance store. This eliminates the EBS I/O wait. * The CPU Shift: Once you remove the disk bottleneck, the task becomes CPU-bound. The CPU is now responsible for the primary key index lookups (Persistent Index on NVMe is fast, but still uses CPU), row merging, and calculating the Delete Vectors (DelVectors). * Verdict: You will see significantly higher throughput in Row Mode on SD+NVMe than on SN+EBS because the worker threads aren't idling on I/O. 2. Why Column Mode "Exploded" Your Compaction The reason you saw compaction scores spike to 2000+ when you disabled Merge Commit for Column Mode is due to Version Bloat: * Row Mode: Each small load creates a small rowset. * Column Mode: Each small load creates a rowset PLUS separate

.cols

files for the updated columns. * The "Double Jeopardy": Without Merge Commit, every small Stream Load is a unique transaction. If you are doing high-throughput small loads, you are creating thousands of "versions" per hour. * Compaction Blockage: In Column Mode, compaction is more complex because it has to merge these delta column files. If your CPUs were already pinned by ingestion, they didn't have enough cycles to perform the heavy lifting of merging those thousands of small column files, leading to the massive rowset buildup you observed. 3. Optimization Recommendations If you are sticking with the Shared Data + NVMe migration: 1. Stick to Row Mode (for now): Since you have the NVMe capacity to ensure 100% cache hits for your TTL, Row Mode will perform exceptionally well. The "Read-before-Write" overhead is significantly mitigated by the local NVMe speed. This is the "safe" path that maintains compatibility with Merge Commit. 2. Monitor

iowait

vs `cpu_usage`: Check your

top

or StarRocks metrics. If

iowait

is near 0% but CPU is 100%, you have successfully shifted the bottleneck. At this point, the only way to scale further is to add more CN nodes or upgrade to higher-vCPU instances (like

i4i.16xlarge

). 3. Check your Version: Compatibility between Column Mode, Merge Commit, and Conditional Update (

merge_condition

) has been improved in v3.3.1+ and v3.4.x. If you are on an older v3.1 or v3.2 build, you are likely hitting the specific bugs (e.g., #65278) that caused your Column Mode test to fail. 4. Persistent Index: Ensure your Persistent Index is stored on the NVMe disk. In Shared Data mode, this is critical for high-throughput partial updates to avoid S3 lookups for the index itself. Summary: Your partial updates will perform better on Shared Data because the NVMe cache turns a remote I/O "Read-before-Write" into a local-speed operation. You are correctly identifying that you will become CPU-bound, but that is a "higher quality" bottleneck that is easier to scale (by adding nodes) than I/O latency. References

@Rocky datacache_disk_path still required definition in format scripts for NVMe disk when starting up EC2 for CN nodes? I read this BE configuration item

datacache_disk_path

is now deprecated. The data will be cached under the directory

${storage_root_path}/datacache

. If you want to allocate a dedicated disk for data cache, you can manually point the directory to the directory mentioned above using a symlink.

Yes, you are correct. Starting from StarRocks v3.4,

datacache_disk_path

is deprecated. The system now automatically manages the data cache within the directory defined by

storage_root_path

. Here is how you should handle your

i4i.8xlarge

setup with multiple NVMe disks: 1. The New Configuration Standard Instead of

datacache_disk_path

, you should now use

storage_root_path

in your

cn.conf

. This parameter supports multiple paths separated by semicolons. Example for

i4i.8xlarge

(2x NVMe disks): If your NVMe disks are mounted at

/data/nvme0

and

/data/nvme1

, your configuration should look like this:

properties
storage_root_path = /data/nvme0;/data/nvme1

StarRocks will automatically create a

datacache

sub-directory under each of these paths (e.g.,

/data/nvme0/datacache

and

/data/nvme1/datacache

) and distribute the cache blocks across them. 2. Why the Symlink Suggestion Exists The documentation mentions the symlink approach for cases where you have a specific architectural requirement to keep the "root" storage directory (which holds small amounts of metadata and logs) on a different physical volume (like a small EBS boot volume) while forcing the heavy cache data onto the NVMe. However, for your high-performance ingestion use case on

i4i

instances, it is generally better to point

storage_root_path

directly to the NVMe mounts. This ensures that both the Data Cache and the Persistent Index (which is critical for your partial update performance) live on the fastest available storage. 3. Key Parameters to Update in Your Scripts When migrating your startup scripts to this new model, ensure you adjust these accompanying parameters: *

datacache_disk_size

: Note that this value now typically applies per disk path defined in

storage_root_path

. If you set it to

1.5TB

and have two paths, your total cache will be

3TB

. _ _Tip:* You can also use a percentage, like

datacache_disk_size = 90%

, which is often safer for ephemeral instance stores. *

enable_datacache_disk_auto_adjust

: (Introduced in v3.3) Set this to

true

(default) to allow StarRocks to dynamically manage the cache space based on actual disk pressure. * Persistent Index: By default, the Persistent Index will also be stored under

storage_root_path

. Since you are using local NVMe, this is exactly what you want for low-latency partial updates. Summary of Action for your Scripts 1. Remove

datacache_disk_path

. 2. Update

storage_root_path

to include all your NVMe mount points (e.g.,

/mnt/nvme0;/mnt/nvme1

). 3. Confirm the directory name: Depending on your exact sub-version, the cache may appear as

datacache/

or

starlet_cache/

. The system handles this automatically, but if you are manually clearing or pre-warming the cache in scripts, check which one your version creates. References • administration/management/BE_configuration.md • using_starrocks/caching/block_cache.md • deployment/shared_data/s3.md • data_source/data_cache_troubleshooting.md References • administration/management/BE_configuration.md • knowledge/usage/best-practice.md • data_source/data_cache_troubleshooting.md • deployment/shared_data/feature-support-shared-data.md • using_starrocks/caching/block_cache.md

@Rocky What exactly is the prometheus metric name for iowait? Does iowait time contribute to CPU utilization too? So with lesser iowait, we reduce context switching or maximize active work when cpu is not idle?

For your monitoring setup on EC2

i4i

nodes, understanding the relationship between

iowait

and CPU is key to tuning your ingestion performance. 1. Prometheus Metric Names Depending on whether you are looking at the host-level metrics (from

node_exporter

) or the StarRocks process-level metrics (from the BE/CN

/metrics

endpoint), the names differ: * StarRocks BE/CN Process Metrics: * Metric:

starrocks_be_cpu{mode="iowait"}

(in some versions exported as

starrocks_be_cpu_iowait

). * Unit: This is usually a counter representing the total CPU time spent in that mode. You should use

irate

or

rate

in PromQL to get the percentage. * Host/Node Metrics (Standard): * Metric:

node_cpu_seconds_total{mode="iowait"}

. * Query for %:

avg by (instance) (irate(node_cpu_seconds_total{mode="iowait"}[5m])) * 100

2. Does

iowait

contribute to CPU utilization? Technically,

iowait

is a sub-category of "Idle" time. * The Definition:

iowait

means the CPU had nothing to do because all runnable tasks were waiting for I/O to complete. * The Calculation: Most standard "CPU Utilization" formulas (like the one in the default StarRocks Grafana dashboard) use

100 - (idle_rate * 100)

. Since

iowait

is not

idle

, it does contribute to the reported "Utilization" percentage in many dashboards. * The Reality: During

iowait

, the CPU core is not actually executing instructions; it is effectively stalled. If you had a purely computational task (like calculating a complex hash) ready to run, the Linux scheduler would give that task the CPU cycles, and

iowait

would disappear in favor of

user

time. 3. Context Switching and "Active Work" By moving to NVMe instance stores and reducing

iowait

, you are making the system significantly more efficient in two ways: 1. Maximizing Active Work: When

iowait

is high, your ingestion threads are frequently entering a "Blocked" state. When you move data to NVMe, those same threads spend less time waiting and more time in the

user

or

system

modes. You aren't necessarily "saving" CPU; you are finally saturating it with real work (merging rows, decompressing blocks) instead of let-it-idle waiting for EBS. 2. Reducing Context Switching Overhead: In a high

iowait

environment, the Linux scheduler frequently has to swap "Blocked" threads out and "Runnable" threads in. This context switching has a non-trivial CPU cost (cache misses, register saves). By keeping the data on local NVMe, your worker threads can often complete their "Read-before-Write" cycle without ever being de-scheduled. This leads to better instruction-per-cycle (IPC) efficiency. Summary for your Migration On your

i4i.8xlarge

nodes, you should expect your

iowait

metric to drop toward zero. Consequently, you will see your

user

and

system

CPU metrics climb. This is the goal: you want your expensive EC2 CPUs to be 90% busy doing actual data processing (

user

time) rather than being 90% "utilized" while actually just waiting for a network-attached disk (

iowait

). References • administration/management/monitoring/metrics.md • administration/management/BE_configuration.md • administration/management/monitoring/Monitor_and_Alert.md • administration/management/monitoring/Manage_Alerts.md References • administration/management/monitoring/metrics.md

@Rocky So in the past 2 days, worst case CPU iowait is ~25-30% at midnight. Does this mean more CPU can be dedicated to compaction? So shared data will most certainly help partial update?

Yes, your observation is spot-on. A 25–30%

iowait

means that for every minute of wall-clock time, your CPU cores are sitting idle for 15–18 seconds simply waiting for EBS to respond. By migrating to Shared Data with NVMe, you are essentially "buying back" that 30% of your CPU's life. Here is how that translates to your compaction and partial update performance: 1. Reclaiming CPU for Compaction In your current Shared Nothing + EBS setup, your CPUs are likely "fighting" for I/O. When a compaction thread tries to merge segments and an ingestion thread tries to perform a "Read-before-Write" for a partial update, both get stuck in

iowait

. When you move to NVMe: * The "Wait" becomes "Work": The 30% of time previously spent in

iowait

will shift to

user

or

system

time. This means your

compact_threads

can now actually execute at full speed. * Faster Completion: Because the I/O bottleneck is gone, a single compaction task that used to take 60 seconds (with 30% I/O wait) might now finish in 40 seconds. This allows the compaction worker to move on to the next tablet in the queue much faster, preventing the "rowset buildup" you saw before. 2. How Shared Data Specifically Helps Partial Updates The "Shared Data" architecture combined with local NVMe cache is a "cheat code" for Primary Key partial updates because of how it handles the Persistent Index and the Read-before-Write cycle: 1. NVMe Persistent Index: In Shared Data mode, the Persistent Index (which tracks which row is in which segment) is stored on the local storage (

storage_root_path

). On NVMe, lookups that were previously high-latency disk hits become near-instant. 2. Cached "Read-before-Write": For a partial update, StarRocks must read the existing row to merge it. In Shared Data, the Data Cache on NVMe ensures that if that row was recently touched (or fits in your TTL), the "Read" part of the update never leaves the NVMe drive. 3. Asynchronous Write to S3: While the final data is eventually persisted to S3, the ingestion "Apply" process is heavily focused on the local cache and index. This decouples the "latency-sensitive" part of the update from the "throughput-heavy" part of object storage. 3. Will it solve the "Compaction Score 2000" issue? Most likely, yes, provided you use Row Mode (which you've already found more stable). * In your previous test, Column Mode without Merge Commit was creating a "version explosion" that even NVMe would struggle to keep up with. * In Row Mode on Shared Data, you can keep Merge Commit enabled. This allows StarRocks to group many small partial updates into a single version. * Since the NVMe ensures the CPU isn't stalled, your compaction threads can keep the "score" low by merging these versions as fast as they arrive. Tuning Tip for the Migration Once you are on NVMe, you can be more aggressive with your compaction settings because you no longer have to worry about "wasting" I/O: *

compact_threads

: Set this to 25% of your total cores (e.g., 8 on an

i4i.8xlarge

). *

lake_compaction_max_tasks

: This FE setting (default

-1

) will automatically scale tasks across your CN nodes. *

max_cumulative_compaction_num_singleton_deltas

: Consider lowering this to

100

(from the default

1000

) to make individual compaction tasks "snappier" and release memory/CPU back to ingestion more frequently. Summary: Your 30%

iowait

is the "smoking gun." Moving to Shared Data + NVMe will eliminate that wait, allowing your CPUs to process compaction tasks significantly faster and preventing the rowset backlog that was killing your partial update performance. References • knowledge/admin_and_operation/shared_data_compaction.md • best_practices/primarykey_table.md

@Rocky I use starrocks_be_disks_total_capacity to monitor EBS usage on my shared nothing cluster. What metric do I use to monitor the NVMe usage per CN instance on shared data cluster?

In a Shared Data (Cloud Native) cluster, monitoring the local NVMe disks on your CN nodes requires a slightly different approach than the one you used for EBS on Shared Nothing. This is because StarRocks treats "Data" and "Cache" as two different categories in its reporting. Here are the metrics you should use: 1. Monitoring Overall NVMe Disk Health (Standard) You can still use the same disk-level metrics to see the total capacity and available space of the NVMe mounts defined in your

storage_root_path

. These metrics report the filesystem-level state of the disks. * Metric:

starrocks_be_disks_total_capacity

* Metric:

starrocks_be_disks_avail_capacity

(The most critical for alerts) * Metric:

starrocks_be_disks_state

(Monitor for

1

to ensure disks are healthy/online) 2. Monitoring the Data Cache (The 3.3+ standard) In a Shared Data cluster, the segments themselves live on S3, so the

starrocks_be_disks_data_used_capacity

metric will often show 0 or a very small value. To see how much of your NVMe is being used for the Data Cache, use these specific metrics: * Metric:

starrocks_be_block_cache_disk_used_bytes

_ _Usage:* This tells you the actual footprint of the cached blocks on your NVMe. * Metric:

starrocks_be_block_cache_disk_quota_bytes

_ _Usage:* This is the limit you set (via

datacache_disk_size

). * Calculated Rate:

starrocks_be_block_cache_disk_used_bytes / starrocks_be_block_cache_disk_quota_bytes

_ _Note:* It is normal for this to reach ~80-90% and stay there. StarRocks only starts evicting data once it hits the threshold (default 80-90%). 3. Monitoring the Persistent Index One important "hidden" consumer of NVMe space on CN nodes is the Persistent Index (used for Primary Key tables). This is stored in the

persist/

folder under your

storage_root_path

and is not counted inside the

block_cache_disk_used_bytes

metric. Since there is currently no dedicated Prometheus metric for "Persistent Index Directory Size," you should monitor it by looking at the gap: * Index Space (Inferred) ≈

(Total Disk Used) - (Block Cache Used)

* Total Disk Used can be calculated as:

starrocks_be_disks_total_capacity - starrocks_be_disks_avail_capacity

. Recommended Dashboard Setup For your

i4i.8xlarge

nodes, I recommend setting up two specific alerts: 1. NVMe Disk Full Alert: Fire when

starrocks_be_disks_avail_capacity / starrocks_be_disks_total_capacity < 0.1

(10% remaining). This protects you from the Persistent Index growing too large and crashing the node. 2. Cache Hit Rate: Monitor

starrocks_be_block_cache_hit_ratio

or

irate(starrocks_be_block_cache_hit_bytes)

vs

miss_bytes

. If this is low while your disk is full, it means your hot data set is larger than your NVMe cache. References • administration/management/monitoring/metrics.md • using_starrocks/caching/block_cache.md • data_source/data_cache_observe.md • administration/management/monitoring/Monitor_and_Alert.md References • using_starrocks/caching/block_cache.md • administration/management/monitoring/metrics.md • data_source/data_cache_observe.md

@Rocky What metric do i use to monitor IO util of the NVMes?

To monitor the I/O utilization of your NVMes, you should look at both the StarRocks-specific metrics and the host-level metrics. 1. StarRocks Internal Metrics StarRocks BE (or CN) processes report their own view of disk performance for the paths defined in

storage_root_path

. * Metric:

starrocks_be_disk_io_util

(sometimes exported as

starrocks_be_max_disk_io_util_percent

) * Description: This represents the percentage of time the disk was busy. * Usage: A value of

0.30

means 30% utilization. In many default StarRocks Grafana templates, this is displayed as a gauge. * Why use it: This specifically monitors the disks StarRocks is using for its local storage (Persistent Index and Data Cache). 2. Host-Level Metrics (Node Exporter) Since you are on

i4i

instances with local NVMe, the most accurate "real-time" view of the hardware performance comes from the Prometheus

node_exporter

. * Metric:

node_disk_io_time_seconds_total

* PromQL for % Util:

promql
    rate(node_disk_io_time_seconds_total{device=~"nvme.+"}[5m]) * 100

* What this tells you: This is the equivalent of the

%util

column in

iostat

. It tells you what percentage of the time there were I/O requests issued to the NVMe device. 3. Key NVMe Performance Metrics (Latency & IOPS) On NVMe drives, Utilization (%util) is often a misleading metric. Because NVMe drives are highly parallel (they have multiple queues), a drive can be "100% utilized" according to Linux but still have plenty of performance headroom. For your migration to Shared Data, monitor these two instead to see if the NVMes are actually "bottlenecked": * Read Latency:

rate(node_disk_read_time_seconds_total[5m]) / rate(node_disk_reads_completed_total[5m])

_ _Target:* For local NVMe, this should be < 0.1ms (or 100 microseconds). If it spikes to 1-2ms, your disks are saturated. * Write Latency:

rate(node_disk_write_time_seconds_total[5m]) / rate(node_disk_writes_completed_total[5m])

Summary Recommendation * For your Dashboard: Use

starrocks_be_disk_io_util

to see if StarRocks thinks the disks are busy. * For Troubleshooting: Use

node_disk_read_time_seconds_total

to check actual hardware latency. If your latency is low (<0.2ms) but

iowait

is high, the issue is usually thread contention, not the physical NVMe speed. References • administration/management/monitoring/metrics.md • administration/management/monitoring/Monitor_and_Alert.md • administration/management/BE_configuration.md References • administration/management/monitoring/metrics.md • administration/management/monitoring/Monitor_and_Alert.md

@Rocky Is it recommended to still use cloud native persistent PK indexes if I have plenty of NVMe rooom?

Yes, even with plenty of NVMe space, it is strongly recommended to use the

CLOUD_NATIVE

persistent index type for your Shared Data cluster. While

LOCAL

might seem attractive because you have the storage overhead to spare,

CLOUD_NATIVE

is the modern standard for StarRocks Shared Data (Cloud Native) for several critical operational and performance reasons. 1. Elasticity and Rebalancing (The "Rebuild" Problem) This is the most important factor. *

LOCAL

Index: The index files are tied to the specific CN node's local disk. If you scale your cluster (add/remove nodes) or if StarRocks decides to rebalance shards to different nodes, the index data does not move with the data. The destination node must rebuild the entire index from scratch by downloading and scanning all data segments from S3. On a large table, this can take hours and saturate your CPU/Network. *

CLOUD_NATIVE

Index: The "Source of Truth" for the index is stored on S3. When a shard moves to a new node, the new node simply pulls the index metadata and begins "lazy loading" the index into its NVMe cache. There is zero rebuild time, making your cluster truly elastic. 2. Performance: Cache vs. Storage You aren't "losing" the speed of your NVMe by choosing Cloud Native. * The

CLOUD_NATIVE

index uses your NVMe as a cache. * Because you have plenty of room, you can ensure your NVMe cache is large enough to hold the entire index. In this scenario, the read performance (lookups for partial updates) is identical to the

LOCAL

index because both are hitting the same NVMe hardware. * The only difference is that

CLOUD_NATIVE

asynchronously flushes index segments to S3 to ensure durability and mobility. 3. Safety and Recovery If an AWS instance fails or the NVMe drive encounters an error: * With

LOCAL

, you lose the index. StarRocks has to re-generate it, which consumes significant resources and can slow down ingestion during the recovery period. * With

CLOUD_NATIVE

, the index is safely on S3. You can replace the node, and it will be back to full performance as soon as the cache warms up. How to best use your "Plenty of NVMe Room" Instead of using the

LOCAL

index type, leverage your extra space to maximize the performance of the

CLOUD_NATIVE

index: 1. Allocate Large Data Cache: Ensure

datacache_disk_size

is set high. Since the Cloud Native index shares the cache infrastructure (in StarRocks 3.3+), having more room ensures that index blocks are rarely evicted. 2. Monitor Cache Hit Rate: Watch the

starrocks_be_block_cache_hit_ratio

. With plenty of NVMe, your goal should be >99% for index-related lookups. 3. Metadata Cache: Ensure your metadata cache settings are tuned so that the FE/CN can keep the index "hot" in memory/cache. Summary Recommendation Stick with

CLOUD_NATIVE

. It gives you the performance of NVMe (via caching) without the fragility of local-only storage.

sql
-- Recommended for your Shared Data environment
PROPERTIES (
    "enable_persistent_index" = "true",
    "persistent_index_type" = "CLOUD_NATIVE"
);

References • best_practices/primarykey_table.md • table_design/table_types/primary_key_table.md • deployment/shared_data/feature-support-shared-data.md • administration/management/BE_configuration.md References • best_practices/primarykey_table.md • deployment/shared_data/feature-support-shared-data.md