@Rocky when will hnsw index for shared data be available

Currently, Hierarchical Navigable Small World (HNSW) indexes are only supported in shared-nothing clusters (starting from StarRocks v3.4). For shared-data (compute-storage separation) clusters, the feature is officially listed as "*To be supported*" in the StarRocks documentation and roadmap. While a specific release version for standard StarRocks clusters has not been publicly committed, here is the current status of related "search" features for shared-data mode: * HNSW/IVFPQ Indexes: Currently restricted to shared-nothing mode. * Full-text Inverted Indexes: Introduced for shared-data mode as a Beta feature in v4.1. * Managed Services: On specific platforms like Alibaba Cloud EMR Serverless StarRocks, vector search support in storage-compute separation has been available in certain versions (v3.5.16+), but this may differ from the standard open-source release. Summary of Roadmap: StarRocks is moving toward a unified "All-in-one" architecture where shared-data becomes the default and includes all search capabilities. Since inverted index support was prioritized for v4.1, vector index (HNSW) support for shared-data is expected in a subsequent 4.x release. For the most up-to-date timeline, it is recommended to keep an eye on the StarRocks GitHub Roadmap or the

#general

channel in the StarRocks community Slack. References • table_design/indexes/vector_index.md • quick_start/shared-data.md • faq/shared_data_faq.md • deployment/shared_data/feature-support-shared-data.md • data_source/data_cache.md

@Rocky what is the performance diff between shared data and shared nothing on querying s3 iceberg tables and materialized views generated from external iceberg catalog

When comparing Shared-data (compute-storage separation) and Shared-nothing architectures for querying S3 Iceberg tables and their Materialized Views (MVs), the performance differences are driven by data locality, caching mechanisms, and architectural optimizations. 1. Querying External Iceberg Tables (S3) For direct queries against external catalogs (Iceberg on S3), both architectures function as a "compute-on-lake" engine. However, there are subtle differences: * Caching Performance: Both architectures rely on the Data Cache to avoid repeated S3 I/O. In Shared-data mode, the Data Cache is often more deeply integrated into the lifecycle of the compute nodes (CN). In official StarRocks benchmarks (SSB 1TB), Shared-data with Data Cache enabled was found to be approximately 1.2x faster than a standard Shared-nothing cluster due to specific scan and I/O merge optimizations optimized for the cloud-native engine. * Metadata Planning: Both use the same distributed metadata parsing for Iceberg (manifest file parallel reading). There is no inherent performance penalty for metadata planning in Shared-data mode. 2. Materialized Views (MVs) Generated from Iceberg When you create an MV on an Iceberg table, StarRocks stores the pre-computed results as an "internal" table. This is where the architectures diverge: Feature | Shared-nothing (Local BE) | Shared-data (Cloud Native) MV Data Storage | Local SSD/HDD on BE nodes. | S3/Object Storage. Read Latency | Very low (Local Disk speed). | Variable (S3 speed), but effectively Local-speed when cached. Scaling | Adding nodes requires data rebalancing (reshuffling MV data). | Instant scaling. New nodes can immediately query MV data by pulling from S3 into their local cache. Cost-Performance | High performance, higher storage cost (SSD). | Lower storage cost; performance matches local disk once the "hot" MV data is cached. Summary of Performance Differences * Cold Start: Shared-nothing is generally faster for the very first query of a Materialized View because the data is already on local disks. Shared-data must fetch the MV data from S3 to populate the cache. * Warm Queries (Cached): Once data is cached, Shared-data matches or exceeds Shared-nothing performance. The Shared-data engine (v3.x+) includes "Parallel Scan" and "I/O Merge" optimizations that are specifically tuned for the compute-storage separation path. * External Queries: For direct Iceberg queries (not MVs), there is virtually no difference if both have Data Cache enabled, though Shared-data is the primary focus for these performance optimizations. Recommendation: * If your workload is highly elastic (e.g., you need to spin up/down compute for reporting), Shared-data is superior and generally faster in throughput due to its newer scan optimizations. * If you have a stable, fixed-size cluster and want the absolute lowest latency for the first query of an MV without relying on cache warming, Shared-nothing is more predictable. References • deployment/shared_data/feature-support-shared-data.md • data_source/catalog/iceberg/iceberg_catalog.md • using_starrocks/async_mv/use_cases/data_lake_query_acceleration_with_materialized_views.md • deployment/shared_data/azure.md • using_starrocks/async_mv/feature-support-asynchronous-materialized-views.md References • deployment/shared_data/feature-support-shared-data.md

@Rocky for shared nothing, how is materialized view replicated onto multiple nodes? Will a primary node for a segment of data will have replicas on other nodes that can help serve requests if the primary is under load?

In the Shared-nothing architecture, StarRocks treats Materialized Views (MVs) similarly to regular tables in terms of data distribution and replication. 1. How MVs are Replicated The replication mechanism depends on the type of Materialized View: * Asynchronous MVs: These are effectively "internal tables" managed by StarRocks. When you create an asynchronous MV, it has its own data distribution (buckets) and replication strategy. You can specify the number of replicas using the

replication_num

property (e.g.,

PROPERTIES("replication_num" = "3")

). If not specified, it defaults to the cluster's default (usually 3). * Synchronous MVs: These are stored as "rollups" or internal indexes physically attached to the base table. They automatically inherit the exact same replication and distribution properties as the base table. If the base table has 3 replicas, the synchronous MV will also have 3 replicas stored on the same Backend (BE) nodes. 2. Primary vs. Secondary Replicas and Load Serving Unlike some systems where only a "leader" can serve reads, StarRocks uses an "Any-Replica-Read" model. * No "Primary" for Reads: For a specific segment of data (a "tablet"), all replicas are considered equal for serving query requests once a transaction is committed. There is no concept of a "Primary node" that must handle all reads for that segment. * Load Balancing by the FE: When a query arrives, the Frontend (FE) acts as the coordinator. It generates an execution plan and decides which BE will handle each tablet. The FE uses a scheduling algorithm (typically a combination of consistent hashing and health checks) to distribute the workload. * Helping with Load: If a BE node is under heavy load or performing poorly, the FE can route requests for that data to one of the other healthy replicas on different nodes. This allows the cluster to: 1. Parallelize queries: Different concurrent queries can read from different replicas of the same tablet. 2. Ensure High Availability: If one node fails or is extremely slow, the FE immediately switches to an alternative replica. * CBO Optimization: You can enable

enable_local_replica_selection

(default:

false

) in the FE configuration. If set to

true

, the Cost-Based Optimizer (CBO) will prefer replicas on the same node as the FE (to reduce network overhead), but in high-load scenarios, it will still distribute tasks to other BEs to maintain performance. Summary In Shared-nothing mode, your Materialized View is replicated across multiple nodes (default 3). All these replicas are active and can serve requests. If one node hosting a segment is under load, the system's coordinator (FE) can utilize the other replicas on different nodes to balance the query pressure and ensure low latency. References • introduction/Architecture.md • administration/management/resource_management/Replica.md • sql-reference/sql-statements/materialized_view/CREATE_MATERIALIZED_VIEW.md • administration/management/FE_configuration.md • table_design/data_distribution/Data_distribution.md References • administration/management/resource_management/Replica.md • administration/management/FE_configuration.md • administration/management/resource_management/Load_balance.md

@Rocky Does shared nothing support async mv with query rewrite and incremental refresh

Yes, Shared-nothing (and Shared-data) architectures fully support asynchronous Materialized Views (MVs) with query rewrite and incremental refresh. In fact, these features were originally developed and matured in the Shared-nothing architecture. 1. Query Rewrite Asynchronous MVs in StarRocks are designed for transparent query acceleration. The Cost-Based Optimizer (CBO) can automatically rewrite queries to use the MV instead of the base tables, even if the query does not explicitly reference the MV. * Supported Rewrites: Includes Single-table, Inner/Outer/Semi/Anti Joins, Aggregations (Sum, Count, Avg, etc.), UNION, and Nested MVs. * Control: You can enable/disable rewrite for specific MVs using the property

"enable_query_rewrite" = "true"

. * Staleness: You can control how "fresh" the MV must be for a rewrite to occur using the

mv_rewrite_staleness_second

property. 2. Incremental Refresh (Partition-level) StarRocks achieves "incremental" performance for async MVs primarily through Partition-level Refresh. Instead of recalculating the entire dataset, the system identifies which partitions in the base table have changed and only refreshes the corresponding partitions in the MV. * How it works: When you define an MV with

PARTITION BY

, StarRocks tracks the version/last-modified-time of the base table partitions. * Triggering: * Internal Tables: Can be triggered automatically as soon as data in the base table changes. * External Tables (Iceberg/S3): Does not support automatic triggers on data change. You must use a fixed-interval schedule (e.g.,

REFRESH ASYNC EVERY(INTERVAL 1 HOUR)

) or a manual trigger (

REFRESH MATERIALIZED VIEW mv_name

). * Efficiency: Even when triggered on a schedule, StarRocks will only refresh the specific partitions that are out-of-sync with the source Iceberg/S3 data, making it functionally incremental. Key Limitations to Keep in Mind: * Non-partitioned MVs: If the MV is not partitioned, any refresh (manual or scheduled) will perform a Full Refresh (re-executing the entire query). * External Data Tracking: For Iceberg tables, StarRocks uses the snapshot ID or partition stats to detect changes. For other external sources like Hive, it may rely on partition file modification times. Summary Table for Shared-nothing Async MVs Feature | Supported | Notes Query Rewrite | Yes | Fully supported since v2.5; supports complex joins/aggs. Incremental Refresh | Yes | Achieved via partition-level updates. Internal Table Source | Yes | Supports automatic "on-commit" refresh. External Table Source | Yes | Manual or Scheduled refresh only; partition-aware. References • using_starrocks/async_mv/feature-support-asynchronous-materialized-views.md • using_starrocks/async_mv/use_cases/query_rewrite_with_materialized_views.md • sql-reference/sql-statements/materialized_view/CREATE_MATERIALIZED_VIEW.md • using_starrocks/async_mv/use_cases/data_lake_query_acceleration_with_materialized_views.md References • using_starrocks/async_mv/feature-support-asynchronous-materialized-views.md