The choice to use a native segment format for shared-data (cloud-native) clusters, rather than a standard open format like Apache Iceberg, is driven by the need for performance parity with local-disk-based databases and the requirements of StarRocks' high-frequency update engine. Here is the breakdown of why this distinction exists: 1. Vectorized Engine Optimization StarRocks' native format (segment files) is co-designed with its vectorized execution engine. * Indexing: Internal tables support specific indexes (Primary Key indexes, Bitmap indexes, and Zone Maps) that allow the engine to skip data at a much more granular level than the standard Parquet files used by Iceberg. * I/O Efficiency: In shared-data mode, StarRocks implements "I/O Merge" and "Parallel Scan" optimizations that are specifically tuned for its segment format. Benchmarks show that shared-data clusters using the native format are typically 1.4x to 2x faster than querying the same data via a Hive or Iceberg catalog without cache. 2. Real-Time Updates (Primary Key Model) A key feature of StarRocks is its Primary Key table model, which supports high-concurrency UPSERTs. * This model relies on Delete Vectors and a specialized metadata service (StarOS) to manage data versions and compaction in real-time. * Implementing this level of update performance on top of Iceberg’s manifest-based snapshot system would introduce significant latency and overhead, making it unsuitable for the sub-second ingestion-to-query latency that StarRocks users expect. 3. Metadata Management vs. Interoperability * Shared-Data (Native): StarRocks acts as the "source of truth." It manages the lifecycle of the data, including automated compaction and vacuuming of old versions on S3. This tight control allows the Frontends (FE) to plan queries much faster because they don't have to parse external manifest files for every query. * External Catalog (Iceberg): This is designed for interoperability. If your goal is to share data with Spark, Trino, or Flink, you should use an Iceberg Catalog. StarRocks supports writing to Iceberg tables via

INSERT INTO

, but it treats them as external because it cannot guarantee the same performance optimizations (like the Primary Key index) that it can with its own format. 4. Can you use Iceberg instead? If you prefer the openness of Iceberg over the performance of the native format, you can still achieve a "shared-data" experience: 1. Deploy a StarRocks cluster (shared-data or shared-nothing). 2. Create an Iceberg Catalog pointing to your S3 bucket. 3. Use

INSERT INTO iceberg_catalog.db.table SELECT ...

to store your data there. Summary: The native format is for speed and advanced database features (like real-time updates), while the Iceberg catalog is for ecosystem compatibility. By offering both, StarRocks lets you choose the right tool for each specific workloads requiring either maximum performance or maximum openness. References • deployment/shared_data/feature-support-shared-data.md • data_source/catalog/iceberg/iceberg_catalog.md • deployment/shared_data/s3.md References • release_notes/release-3.3.md • data_source/feature-support-data-lake-analytics.md

This is a common point of confusion. To clarify: the StarRocks storage format is open source (licensed under Apache 2.0), but it is not an industry standard specification like Apache Iceberg. Here is the difference between being "Open Source" and being an "Open Standard": 1. The Code is Open, the Format is "Internal" Because StarRocks is open source, the code that defines how data is written to disk (segment files) is publicly available on GitHub. Anyone can read, modify, or build a reader for it. However, unlike Iceberg, which has a strict community-governed specification and SDKs for many languages (Java, Python, Rust), the StarRocks format is an internal implementation detail designed specifically for the StarRocks engine. 2. Performance vs. Interoperability The primary reason StarRocks doesn't use Iceberg as its only format is the trade-off between speed and compatibility: * Iceberg (Open Standard): Designed to be the "lowest common denominator" so that Spark, Trino, Flink, and StarRocks can all read it. To maintain this compatibility, it cannot easily include engine-specific optimizations. * StarRocks Native (Optimized): Designed to be the "highest possible performance." It includes specialized Short-key indexes, Bitmap indexes, and Delete Vectors (for real-time updates) that are tightly coupled to StarRocks' vectorized execution engine. If StarRocks moved to a universal standard for its internal storage, it would likely lose the 2x–5x performance edge it has over general-purpose engines. 3. "Openness" is about choice StarRocks' philosophy is to provide Open Lakehouse capabilities by supporting both: * For Performance: Use the native format. It gives you the fastest BI and real-time UPSERTs. * For Openness: Use the Iceberg Catalog. StarRocks allows you to create, read, and even write (

INSERT INTO

) to Iceberg tables. This ensures your data isn't "locked in" and can be read by any other tool in your ecosystem. Summary The format isn't "closed"—it's just engine-optimized. * Iceberg is a "bridge" between different tools. * StarRocks Native is the "engine room" designed for maximum speed. By being open source, StarRocks ensures you can see how it works; by supporting Iceberg, it ensures you can take your data wherever you want. References • introduction/what_is_starrocks.md • data_source/catalog/iceberg/iceberg_catalog.md • integrations/data_lakes.mdx • deployment/shared_data/feature-support-shared-data.md References • data_source/catalog/iceberg/iceberg_catalog.md • introduction/what_is_starrocks.md • integrations/data_lakes.mdx • deployment/shared_data/feature-support-shared-data.md References • integrations/data_lakes.mdx • introduction/Features.md • data_source/data_lakes.mdx • faq/Others.md • data_source/feature-support-data-lake-analytics.md • data_source/catalog/iceberg/iceberg_catalog.md

To achieve sub-100ms response times in shared-data (cloud-native) mode, you should leverage StarRocks' multi-layer caching and pre-aggregation features. Even though the data is durably stored in S3, these features ensure that hot data is served from local NVMe or memory. 1. For Alerts (High-Frequency Point Lookups) Alerts typically involve looking up a specific row or status by an ID. * Primary Key Model: Always use the Primary Key table for alerts. It is designed for real-time updates and fast lookups. * Persistent Index: In shared-data mode, ensure the Persistent Index is enabled. Set it to

LOCAL

to keep the primary key index on the local disk of your Compute Nodes (CNs). This avoids network calls to S3 for index lookups. * Row-Columnar Storage (v3.2+): Enable row-columnar storage for these tables. This allows StarRocks to retrieve an entire row in a single I/O operation (sub-1ms), rather than scanning multiple columns.

sql
    PROPERTIES (
        "enable_persistent_index" = "true",
        "persistent_index_type" = "LOCAL",
        "column_context" = "{\"common_column_group\":\"pb\"}"
    )

2. For Dashboards (Aggregation & ETL) Dashboards usually involve

SUM

,

COUNT

, and

GROUP BY

operations. * Synchronous Materialized Views (v3.4+): If your dashboards query specific aggregates, use a Synchronous MV. These are now supported in shared-data mode. They pre-calculate the results during data ingestion, reducing query time from seconds to milliseconds. * Query Cache (v3.4+): Enable the Query Cache. It stores the final or intermediate result sets in the CN's memory. If the same dashboard is viewed by multiple users or refreshed frequently, the results are served instantly from memory.

sql
    SET [GLOBAL] enable_query_cache = true;

3. Data Cache "Pinning" & Warmup To ensure your ETL results never wait for S3 I/O: * Data Cache: Ensure

datacache.enable = true

is set (default). * Infinite TTL: Set

datacache.partition_duration

to a very long time (e.g.,

100 YEAR

) so your "hot" dashboard data is never evicted from local SSDs to make room for older data. * Proactive Warmup: After your ETL job completes, run a

CACHE SELECT

query. This pulls the newly processed data from S3 into the local CN cache immediately, so the first dashboard user experiences sub-100ms speeds.

sql
    CACHE SELECT * FROM your_dashboard_etl_table WHERE report_date = '2023-10-27';

4. Health Check: Compaction Score High latency in shared-data mode often comes from insufficient compaction (too many small files on S3). * Monitor the Compaction Score using

SHOW PROC '/compactions'

. * If the score is consistently > 10, your queries will slow down as they have to merge many files. You can increase

compact_threads

in the BE/CN configuration to speed up this process. Summary Use Case | Core Optimization | Target Latency Alert Point Lookups | PK Table + Row Store + Local Persistent Index | < 10ms Dashboard Aggregates | Synchronous MV + Query Cache | < 50ms First-load Performance | Data Cache Warmup (

CACHE SELECT

) | < 100ms References • table_design/table_types/primary_key_table.md • using_starrocks/caching/query_cache.md • deployment/shared_data/feature-support-shared-data.md • using_starrocks/Materialized_view-single_table.md • faq/shared_data_faq.md References • table_design/hybrid_table.md

• sql-reference/sql-statements/table_bucket_part_index/CREATE_TABLE.md

To achieve sub-100ms response times in shared-data mode (cloud-native), you don't need to bypass S3. StarRocks is designed to provide "local disk speed" for hot data while using S3 for cost-effective durability. For your alerts and dashboard use cases, here is how you configure your tables to hit that <100ms target: 1. For "Alerts": Primary Key Table + Row-Store If your alerts involve high-concurrency point lookups (e.g., fetching a specific alert by ID), use the Primary Key model with Row-oriented storage (available since v3.2.3). StarRocks is natively columnar (fast for analytics), but the row-store property adds a secondary row-based index that allows for sub-10ms point lookups, even in shared-data mode. * Key Property: Set

column_context

to enable the row-store (internal optimized format). * Persistent Index: In shared-data mode, keep the primary key index on local disks for maximum speed.

sql
    PROPERTIES (
        "enable_persistent_index" = "true",
        "persistent_index_type" = "LOCAL", -- Keep the "address book" on local NVMe
        "column_context" = "{\"common_column_group\":\"pb\"}" -- Enables row-oriented optimization
    );

2. For "Dashboards": Data Cache Pinning For dashboard ETL tables (aggregations like

SUM

,

COUNT

,

GROUP BY

), you need to ensure the data is "pinned" to the local NVMe cache of your compute nodes so the query never waits on S3. * Write-Through Cache: In StarRocks, when you load data, it is written to S3 and the local disk cache simultaneously by default. Your first query will hit the local disk, not S3. * Long TTL: Ensure your dashboard data stays "hot" by setting a long partition duration.

sql
    PROPERTIES (
        "datacache.enable" = "true",
        "datacache.partition_duration" = "1 YEAR" -- Effectively pins data to local disks
    );

3. Sub-100ms "Insurance": Cache Warmup If your ETL process finishes and you want to guarantee that the very first user who opens the dashboard gets a sub-100ms response, use the Cache Warmup feature (introduced in v3.3). After your ETL

INSERT

completes, run:

sql
CACHE SELECT * FROM dashboard_etl_table;

This proactively pulls any data not already in the local NVMe cache from S3. You can even schedule this as a routine task to run immediately after your ETL pipeline finishes. Summary of the "Warm" Path Requirement | Solution | Speed Point Lookups (Alerts) | Primary Key + Row-Store | < 10ms Dashboard Queries | Data Cache (NVMe) | < 100ms Data Durability | S3 (Cloud-Native Storage) | (Background) By combining the Data Cache (for the data itself) and the Persistent Index (for the primary keys), you get the sub-100ms performance of a local cluster while keeping the architectural benefits of shared-data (instant scaling, lower cost). References • deployment/shared_data/feature-support-shared-data.md • table_design/table_types/primary_key_table.md • data_source/block_cache_warmup.md • introduction/what_is_starrocks.md • best_practices/primarykey_table.md References • table_design/hybrid_table.md • deployment/shared_data/feature-support-shared-data.md

To achieve sub-100ms response times in shared-data (cloud-native) mode, you must rely on StarRocks' multi-layer caching and indexing architecture. While shared-data clusters store the "source of truth" on S3, the "execution" can be entirely local. Here is the specific configuration strategy for your use cases: 1. The Alerts Table (High Concurrency & Point Lookups) For alerts, you typically need to retrieve a full row based on an ID. In shared-data mode, the Primary Key (PK) Table is your best tool. * Persistent Index on Local NVMe: In shared-data mode, the index that maps keys to data locations can be kept on the local SSD of your Compute Nodes (CN). This avoids S3 lookups during updates and queries. * Property:

"enable_persistent_index" = "true"

and

"persistent_index_type" = "LOCAL"

(v3.1.4+). * Limitation Note: The "Hybrid row-column" storage (which provides a row-store for sub-10ms lookups) is currently not supported in shared-data mode. To hit sub-100ms for alerts in shared-data mode, you must ensure your CN nodes have high-speed NVMe/SSD and sufficient Data Cache to keep the alert table's segment files local. 2. Dashboard ETL Tables (Analytical Speed) Dashboards usually involve aggregations (

GROUP BY

,

SUM

). To make these sub-100ms, you must ensure the data never leaves the CN node's local storage. * Pinning Data to Cache: Set a long duration for the cache to prevent S3 round-trips.

sql
    PROPERTIES (
        "datacache.enable" = "true",
        "datacache.partition_duration" = "1 YEAR"
    )

* Proactive Warming (v3.3+): Immediately after your ETL job (e.g., an

INSERT INTO ... SELECT

) finishes, trigger a Cache Warmup. This "pushes" the new data into the local NVMe cache so the very first dashboard user gets a sub-100ms response.

sql
    -- Warm up the specific data needed for the dashboard
    CACHE SELECT * FROM dashboard_table WHERE report_date = '2024-05-20';

3. Critical Infrastructure Setup To guarantee these speeds in shared-data mode, your Compute Nodes (CN) must be configured as follows: 1. Local Storage: CN nodes must have fast local disks (SSD/NVMe). Configure

storage_root_path

in

cn.conf

to point to these disks. 2. Cache Size: Ensure

starlet_star_cache_disk_size_percent

(default 90%) allows enough room to hold your "hot" alert and dashboard data. 3. Unified Cache (v3.4+): If you are on version 3.4 or later, the data lake and internal table caches are unified, making management simpler and more efficient. Summary Strategy Use Case | Table Model | Key Optimization Alerts | Primary Key |

enable_persistent_index

+

LOCAL

index type Dashboards | Duplicate/PK |

datacache.partition_duration

+

CACHE SELECT

ETL Flow | Any | Use

CACHE SELECT

immediately after Load/ETL If your alerts require sub-10ms (single-digit) latency under extremely high concurrency, you might need to consider a Shared-nothing cluster for that specific table to utilize the Hybrid row-column storage (beta), as that optimization is currently exclusive to local storage clusters with local BE local storage. References • table_design/table_types/primary_key_table.md • deployment/shared_data/feature-support-shared-data.md • using_starrocks/caching/block_cache.md • table_design/hybrid_table.md • administration/management/monitoring/metrics-shared-data.md