http://starrocks.io logo
Join Slack
Powered by
<@U0A71G31CDV> im running starrocks with docker on...
# questions-and-troubleshooting
k
@Rocky im running starrocks with docker on a mac apple silicon with this tutorial: docs.starrocks.io/docs/quick_start/shared-nothing im getting this error while creating an external catalog: Query 1 ERROR at Line 1: : connector create failed {Failed to init connector [type: iceberg, name: iceberg]. msg: Failed to load rest catalog}
r
The error 
Failed to load rest catalog
during Iceberg catalog creation usually indicates that the StarRocks Frontend (FE) cannot reach or authenticate with your Iceberg REST server. Since you are running StarRocks in a Docker container on a Mac (Apple Silicon), this is likely a networking or configuration issue. Here are the most common causes and fixes: 1. Networking: Use 
host.docker.internal
 If your Iceberg REST catalog (e.g., Polaris, Tabular, or a local REST fixture) is running on your Mac host (outside the StarRocks container) and you are using 
localhost
or 
127.0.0.1
in the 
iceberg.catalog.uri
, the connection will fail. Inside the container, 
localhost
refers to the container itself. * Fix: Replace 
localhost
with 
host.docker.internal
.
Copy code
sql
    CREATE EXTERNAL CATALOG iceberg
    PROPERTIES (
        "type" = "iceberg",
        "iceberg.catalog.type" = "rest",
        "iceberg.catalog.uri" = "<http://host.docker.internal:8181>", -- Correct for Mac Docker
        "iceberg.catalog.warehouse" = "your_warehouse_name"
    );
2. Mandatory Properties For a REST catalog, StarRocks requires specific properties. If you are missing any, the initialization will fail. Ensure you have: * 
"iceberg.catalog.type" = "rest"
* 
"iceberg.catalog.uri"
(The API endpoint) * 
"iceberg.catalog.warehouse"
(The warehouse identifier) 3. OAuth2 Configuration (If applicable) If your REST catalog requires authentication (like Apache Polaris), ensure your credentials are correct. There is a known issue in some versions where the 
scope
parameter might be required.
Copy code
sql
"iceberg.catalog.security" = "oauth2",
"iceberg.catalog.oauth2.credential" = "client_id:client_secret",
"iceberg.catalog.oauth2.scope" = "PRINCIPAL_ROLE:ALL" -- Often required for Polaris
4. Apple Silicon (ARM64) Image Compatibility Ensure you are using an official StarRocks image that supports ARM64. The 
starrocks/all-in-one:latest
image generally supports ARM64, but if you are using an older version, the Java Native Interface (JNI) calls required for certain external connectors might fail. * Recommendation: Use the 
starrocks/all-in-one:latest
or 
starrocks/fe-ubuntu:latest
images which are regularly tested on Apple Silicon. 5. Check Logs for Details The error message in the SQL client is a summary. To see the underlying Java exception (e.g., 
Connection Refused
or 
401 Unauthorized
), check the FE logs:
Copy code
bash
docker exec -it quickstart tail -f /opt/starrocks/fe/log/fe.warn.log
Look for a stack trace starting with 
com.starrocks.connector.iceberg.rest.IcebergRESTCatalog
. If you are trying to test Iceberg locally, it is highly recommended to use the StarRocks Iceberg Docker Compose instead of the "shared-nothing" (all-in-one) tutorial, as it sets up a pre-configured network with a REST catalog and MinIO for you. References • developers/mac-compile-run-test.md • quick_start/shared-nothing.md • quick_start/iceberg.md
k
@Rocky i am using the same exact create catalog statement as my deployed shared data cluster on eks why woudlnt this work locally on docker:
Copy code
CREATE EXTERNAL CATALOG `iceberg`
PROPERTIES (
"aws.s3.access_key"  =  "**",
"aws.s3.secret_key"  =  "**",
"iceberg.catalog.rest.access-key-id"  =  "**",
"iceberg.catalog.uri"  =  "<https://s3tables.us-east-1.amazonaws.com/iceberg>",
"type"  =  "iceberg",
"aws.s3.use_aws_sdk_default_behavior"  =  "false",
"iceberg.catalog.rest.secret-access-key"  =  "**",
"aws.s3.region"  =  "us-east-1",
"aws.s3.use_instance_profile"  =  "false",
"iceberg.catalog.rest.sigv4-enabled"  =  "true",
"iceberg.catalog.warehouse"  =  "arn:aws:s3tables:us-east-1:**:bucket/**",
"iceberg.catalog.rest.signing-name"  =  "s3tables",
"iceberg.catalog.type"  =  "rest"
)
r
The error 
Failed to load rest catalog
occurs when the StarRocks Frontend (FE) cannot successfully perform the initial handshake (usually a 
GET /v1/config
call) with the Iceberg REST endpoint. Since this works on EKS but fails in your local Docker environment on Mac, the issue is likely related to network connectivity, SSL/Trust stores, or the Docker image version. 1. Match the Official S3 Tables Configuration While AWS S3 Tables has its own endpoint, the StarRocks documentation specifically recommends accessing S3 Tables through the AWS Glue Iceberg REST endpoint. There are slight differences in the signing name and URI. Try updating your catalog statement to match the official guide: * URI: 
<https://glue>.<region>.<http://amazonaws.com/iceberg|amazonaws.com/iceberg>
* Signing Name: 
glue
Copy code
sql
CREATE EXTERNAL CATALOG iceberg
PROPERTIES (
    "type" = "iceberg",
    "iceberg.catalog.type" = "rest",
    "iceberg.catalog.uri" = "<https://glue.us-east-1.amazonaws.com/iceberg>", -- Change to Glue endpoint
    "iceberg.catalog.rest.sigv4-enabled" = "true",
    "iceberg.catalog.rest.signing-name" = "glue", -- Change to glue
    "iceberg.catalog.rest.access-key-id" = "YOUR_ACCESS_KEY",
    "iceberg.catalog.rest.secret-access-key" = "YOUR_SECRET_KEY",
    "iceberg.catalog.warehouse" = "YOUR_ACCOUNT_ID:s3tablescatalog/YOUR_TABLE_BUCKET",
    "aws.s3.region" = "us-east-1"
);
2. Check for SSL/Certificate Issues Local Docker images (especially older 
all-in-one
builds) sometimes lack the updated CA certificates required to verify the SSL for newer AWS endpoints like 
s3tables
. * Test: Check if the container can reach the endpoint using 
curl
from inside the container:
Copy code
bash
    docker exec -it <container_id> curl -v <https://s3tables.us-east-1.amazonaws.com/iceberg/v1/config>
* If you see an SSL certificate error, you may need to update the CA certificates inside the container or provide a custom trust store. 3. Verify Docker Image Version Support for 
iceberg.catalog.rest.sigv4-enabled
and S3 Tables was added in StarRocks v3.3.6+ and v3.4.1+. * The 
starrocks/all-in-one:latest
image on Docker Hub is often a specific version (like 3.2.x or 3.3.x). If your local image is cached and older than the version running on EKS, it won't recognize the SigV4 properties. * Fix: Run 
docker pull starrocks/all-in-one:latest
to ensure you have the most recent version, or explicitly use 
starrocks/all-in-one:3.4.2
(or higher). 4. Debugging via FE Logs The SQL error is a summary. The actual Java exception (e.g., 
UnknownHostException
, 
SSLHandshakeException
, or 
403 Forbidden
) is logged in the FE. Run this to see the detailed stack trace:
Copy code
bash
docker exec -it <container_id> tail -f /opt/starrocks/fe/log/fe.warn.log
Look for lines starting with 
Failed to load rest catalog
. If it says "Unknown Property", your StarRocks version is too old. If it says "Connection Refused", it is a networking/proxy issue. 5. IAM Permissions (Local vs EKS) On EKS, your cluster might have an IAM Role (IRSA) that grants 
s3tables:_
and 
glue:_
permissions. Locally, you are using Access Keys. Ensure that the specific IAM User associated with those keys has: 1. S3 Tables permissions (
s3tables:GetTable
, etc.) 2. Glue permissions (
glue:GetDatabase
, etc.) 3. Lake Formation permissions (if applicable, as S3 Tables requires Lake Formation "Allow external engines" setting). References • data_source/catalog/iceberg/iceberg_rest_s3.md • data_source/catalog/iceberg/iceberg_catalog.md • quick_start/shared-nothing.md ) References • data_source/catalog/iceberg/iceberg_rest_s3.md
k
@Rocky For my shared data cluster, how can I have a mat view based on an iceberg table perform as fast as a Starrocks native table. Is the mat view storage format on disk the same as a BE table storage? Even with a cache query on the mat view it takes much longer for queries to complete than a local single back end node (point queries 200 ms vs 50 ms)
r
To achieve performance for a Materialized View (MV) on Iceberg that matches a native StarRocks table, you need to treat the MV exactly like a native table. In StarRocks, an asynchronous materialized view is a native physical table under the hood, but it may not be configured with the same optimizations as your "50ms" native table by default. Here is why you are likely seeing the 200ms vs. 50ms gap and how to close it: 1. Storage Format & Table Model Yes, the storage format on disk for an asynchronous MV is identical to a StarRocks native table. However, there is a key difference in the Table Model: * Native Table (50ms): If you are doing point queries (searching by a specific ID), your native table is likely a Primary Key table. These use an in-memory primary key index for extremely fast lookups. * Materialized View (200ms): Asynchronous MVs are typically stored as Duplicate Key tables. They do not have the same in-memory primary key index. Point queries on Duplicate Key tables rely on the Prefix Index (sort keys) or other secondary indexes. 2. How to Optimize the MV for Point Queries To get closer to native table performance, you must manually apply native table optimizations to your MV definition: * Define Sort Keys (
ORDER BY
): When creating the MV, use the 
ORDER BY
clause to ensure the column you are querying by is the first column in the sort key. This enables the Prefix Index.
Copy code
sql
    CREATE MATERIALIZED VIEW iceberg_mv
    ORDER BY (your_id_column) -- This is critical for point queries
    DISTRIBUTED BY HASH(your_id_column)
    AS SELECT ... FROM iceberg_catalog.db.table;
* Add Secondary Indexes: Just like a regular table, you can add Bloom Filter or Bitmap indexes to an MV after it is created to accelerate specific filters.
Copy code
sql
    -- Add Bloom Filter for the ID column
    ALTER MATERIALIZED VIEW iceberg_mv SET ("bloom_filter_columns" = "your_id_column");
* Check Bucketing: Ensure the 
BUCKETS
count in your MV matches the native table's distribution. Too many buckets for a small point query can add latency due to scheduling overhead. 3. Understanding the 150ms Overhead The difference between 50ms and 200ms in a "Shared Data" (EKS) environment vs. a "Local BE" is often attributable to: * Query Rewrite Overhead: When you query the base Iceberg table and let the optimizer "transparently rewrite" it to use the MV, there is a slight planning overhead (usually 10-50ms). Try querying the MV directly (
SELECT ... FROM iceberg_mv
) to see if the latency drops. * Metadata Overhead: Shared-data clusters (StarRocks 3.x+) fetch metadata from an object store (S3). Even if the data is cached, the metadata lookup or the "Shared-Data" architecture overhead can account for the extra milliseconds compared to a single local BE node reading from an NVMe SSD. * Data Cache: Ensure the 
Data Cache
is warm. In a shared-data cluster, the first query fetches data from S3 to the BE's local disk/memory cache. Subsequent queries should be faster. Summary Checklist 1. Direct Query: Query the MV name directly to bypass rewrite logic. 2. Sort Keys: Ensure 
ORDER BY
is used in the MV creation for your lookup columns. 3. Index: Add a Bloom Filter to the lookup columns. 4. Table Model: If 50ms is a hard requirement for point queries and the MV (Duplicate model) can't reach it, you may need to use a Primary Key table and an 
INSERT INTO ... SELECT
job instead of an MV. References • using_starrocks/async_mv/use_cases/data_lake_query_acceleration_with_materialized_views.md • sql-reference/sql-statements/materialized_view/CREATE_MATERIALIZED_VIEW.md
k
@Rocky how could I define a Starrocks native table in shared data mode that is periodically refreshed from an external iceberg table
r
To define a native table in a shared-data cluster that periodically refreshes from Iceberg, you have two main options. The choice depends on whether you need the Primary Key table model for high-speed point queries or the simpler Materialized View approach. Option 1: Scheduled Task + Native Primary Key Table (Best for Point Queries) Since you mentioned needing 50ms point-query performance, a Primary Key table is your best choice. Materialized Views currently use the Duplicate Key model, which lacks the in-memory primary index for sub-100ms lookups. You can automate the refresh using StarRocks Tasks (available in v3.3+). 1. Create the native Primary Key table:
Copy code
sql
CREATE TABLE native_iceberg_sync (
    id BIGINT NOT NULL,
    data_val VARCHAR(255),
    updated_at DATETIME
)
PRIMARY KEY (id)
DISTRIBUTED BY HASH(id)
PROPERTIES (
    -- Use cloud-native persistent index for shared-data performance
    "enable_persistent_index" = "true",
    "persistent_index_type" = "CLOUD_NATIVE" 
);
2. Schedule a periodic refresh task: This task will run an 
INSERT INTO ... SELECT
at your desired interval. Because it is a Primary Key table, it will automatically perform an Upsert (updating existing rows and inserting new ones).
Copy code
sql
SUBMIT TASK iceberg_refresh_job
SCHEDULE EVERY(INTERVAL 10 MINUTE)
AS 
INSERT INTO native_iceberg_sync
SELECT id, data_val, updated_at 
FROM iceberg_catalog.your_db.your_table
WHERE updated_at > date_sub(now(), INTERVAL 15 MINUTE); -- Incremental sync example
Option 2: Asynchronous Materialized View (Easiest Management) An Asynchronous MV is essentially a native StarRocks table that StarRocks manages for you. It is the recommended way to accelerate Iceberg data if you don't specifically need the Primary Key model.
Copy code
sql
CREATE MATERIALIZED VIEW iceberg_mv
-- Optimizations for performance matching native tables
ORDER BY (id) 
DISTRIBUTED BY HASH(id)
REFRESH ASYNC EVERY(INTERVAL 10 MINUTE) 
PROPERTIES (
    "datacache.enable" = "true"
)
AS 
SELECT id, data_val, updated_at 
FROM iceberg_catalog.your_db.your_table;
Key Differences for Shared-Data Mode * Storage Format: Both options store data in StarRocks' internal format on your object storage (S3/EKS). * Query Performance: The Primary Key table (Option 1) will be faster for point queries (searching by ID) because of its unique index. The MV (Option 2) is better for analytical scans and aggregations. * Caching: In shared-data mode, ensure 
datacache.enable
is 
true
(it is by default). This ensures that once the data is "refreshed" into the native table, it stays on the local NVMe/SSD of the BE nodes for that 50ms response time. * Incremental Refresh: MVs on Iceberg can automatically detect partition changes and refresh only the new data. With a manual 
TASK
, you must write the 
WHERE
clause logic yourself to avoid scanning the entire Iceberg table every time. Recommendation: If you need the 50ms latency you mentioned previously, use Option 1 with a Primary Key table and a scheduled 
TASK
. References • sql-reference/sql-statements/loading_unloading/ETL/SUBMIT_TASK.md • table_design/table_types/primary_key_table.md • using_starrocks/async_mv/use_cases/data_lake_query_acceleration_with_materialized_views.md References • best_practices/primarykey_table.md • table_design/table_types/primary_key_table.md
k
@Rocky for option 1 if the table is a native primary key tsbw why is it store on s3 and not directly on Starrocks disk?
r
In a Shared Data (StarRocks 3.x+) cluster, the term "Native Table" refers to the data format and management, not the physical storage location. Here is the breakdown of why your Primary Key table lives on S3 and how it still achieves high performance: 1. Storage vs. Compute Separation In Shared Data mode, StarRocks follows a "Cloud Native" architecture where the Source of Truth is always remote object storage (S3). * Persistent Layer (S3): All native tables (including Primary Key tables) store their actual data files (segments) on S3. This allows you to scale your compute nodes (CNs/BEs) up and down instantly without moving data. * Native vs. External: A table is "Native" because StarRocks owns the metadata, the indexing, and the background compaction. An "External" table (like Iceberg) is one where StarRocks just reads a format owned by another system. 2. The Role of the Local Disk (Data Cache) Even though the data is stored on S3, StarRocks uses your local BE/CN disks (NVMe/SSD) as a sophisticated cache layer. * Datacache: When you create a native table with 
"datacache.enable" = "true"
(the default), StarRocks writes data to both S3 and the local disk during ingestion. * Point Queries: For your 50ms requirement, the query doesn't actually go to S3. The BE/CN checks its local cache. If the data is there, it reads from the local NVMe/SSD at local speeds. * Persistent Index: For Primary Key tables specifically, StarRocks also maintains a "Persistent Index." In shared-data mode, this index can also be offloaded and cached to ensure fast lookups without needing massive amounts of RAM. 3. Why not store it directly on disk? If you store data only on the BE disks (the old "Shared Nothing" way), you lose the primary benefits of EKS/Cloud-native deployments: * Scalability: If a BE node dies or you scale down, you lose the data on that disk. With S3 storage, any new BE node can immediately start serving the same data by pulling from S3 into its local cache. * Cost: S3 is significantly cheaper for storing terabytes of data than high-performance EKS EBS volumes or local NVMe. Summary of Performance The reason your "Local Docker" setup is fast is likely because the disk and compute are on the same machine. In your Shared Data EKS cluster, you get the same 50ms performance as long as the Data Cache is warm. If you are seeing 200ms on a native table in EKS, check these two things: 1. Warm the Cache: Is the data actually in the cache? Run the query twice; the second time should be much faster. 2. Persistent Index: Ensure you have 
"enable_persistent_index" = "true"
in your table properties so the Primary Key lookups are optimized for the cloud-native architecture. References • introduction/Architecture.md • deployment/shared_data/s3.md • table_design/table_types/primary_key_table.md References • introduction/Architecture.md • deployment/shared_data/s3.md • table_design/table_design.md • deployment/shared_data/feature-support-shared-data.md
k
@Rocky if I do option 1 will we avoid iceberg metadata planning
r
Yes, Option 1 (Primary Key table) completely avoids Iceberg metadata planning. When you move data into a native StarRocks table, you are switching from the "Data Lake" metadata model to the "Native Database" metadata model. Here is the technical difference and why it fixes your latency issue: 1. Iceberg Metadata Planning (The 200ms Path) Even with caching, querying an Iceberg table requires the StarRocks FE (Front End) to: * Check Snapshots: Determine which Iceberg snapshot is current. * Parse Manifests: Read and parse manifest files (Avro format) to find which data files match your query filters. * S3 Coordination: Even if manifest files are cached on the BE, the FE still spends time coordinating the "Plan" across the cluster to figure out where the data lives. This "Planning" phase often accounts for a significant chunk of that 200ms latency, especially in shared-data/EKS environments where network overhead is higher than a local Docker container. 2. Native Table Planning (The 50ms Path) For a native StarRocks table, the metadata is managed internally: * Direct Tablet Lookup: The FE keeps the mapping of data (tablets) in its own memory. When you query by an ID, the FE immediately knows which BE node and which specific "Segment" file contains that ID. * Zero Parsing: There are no manifest files or Avro files to parse. The planning happens in a few milliseconds because it is a simple memory lookup in the StarRocks catalog. * Primary Key Index: In Option 1, the Primary Key Index acts as a shortcut. Instead of scanning files, the BE uses its in-memory index to jump directly to the exact row. Summary of Benefits for Option 1: Feature | Iceberg External Table / MV | Native Primary Key Table (Option 1) Metadata Source | Iceberg Manifests (S3/Cache) | Internal FE Catalog (In-Memory) Planning Time | High (50ms - 150ms+) | Low (<5ms - 10ms) Point Query Speed | Slow (Scan-based) | Instant (Index-based) Refresh Logic | Managed by MV (Transparent) | Managed by Task (You control it) Conclusion: By syncing the data to a native table, you shift the "heavy lifting" of Iceberg metadata parsing to your background refresh task. Your user-facing queries only interact with the ultra-lean native StarRocks metadata, which is how you achieve the 50ms response time. References • data_source/catalog/iceberg/iceberg_catalog.md • introduction/Architecture.md • table_design/table_types/primary_key_table.md References • data_source/catalog/iceberg/iceberg_catalog.md • data_source/feature-support-data-lake-analytics.md
k
@Rocky does Starrocks support primary key row storage for shared data mode
r
As of the current version (v3.3), StarRocks does not support persistent hybrid row-column storage (row store) in shared-data mode. While you can create a Primary Key table in shared-data mode, there is a distinction between the update mode and the physical storage format: 1. Hybrid Row-Column Storage (The "Row Store" feature) This feature (enabled by 
"store_row_column" = "true"
) creates a dedicated row-oriented storage alongside the columnar data to accelerate point queries and short-circuit lookups. * Shared-Nothing (Local Storage): Supported. * Shared-Data (S3 Storage): Not supported ❌. According to the official compatibility matrix, this is planned for a future release but is currently unavailable in shared-data clusters. 2. Partial Update in "Row Mode" You might see "Row mode" mentioned in documentation for shared-data clusters. This refers to the Partial Update strategy: * Row Mode (Supported): When you update a few columns, StarRocks reads the entire row, updates the necessary fields, and writes a completely new version of the row. This is the default and only way shared-data clusters handled partial updates until recently. * Column Mode (Supported since v3.3.1): Only the updated columns are written, which is more efficient for tables with hundreds of columns. Impact on your 50ms Goal Even without the dedicated "Row Store" property, you can still achieve very low latency in shared-data mode by following these steps: 1. Use Primary Key Tables: They use a "Delete+Insert" strategy that is significantly faster for point queries than other models. 2. Persistent Index: Use 
"enable_persistent_index" = "true"
and 
"persistent_index_type" = "CLOUD_NATIVE"
. This ensures the primary key index (which maps your ID to the file location) is stored on S3 and cached locally, preventing high memory usage while keeping lookups fast. 3. Data Cache: Ensure your EKS nodes have sufficient NVMe/SSD space for the 
datacache
. Since your queries are likely point queries on "hot" data, the data will be served from the local cache, avoiding S3 latency. Summary: You cannot use the 
store_row_column
property yet in your EKS shared-data environment, but a standard Primary Key table with a warm Data Cache is usually sufficient to meet sub-100ms requirements. References • deployment/shared_data/feature-support-shared-data.md • table_design/hybrid_table.md • table_design/table_types/primary_key_table.md References • deployment/shared_data/feature-support-shared-data.md • table_design/table_types/primary_key_table.md • table_design/hybrid_table.md
k
@Rocky what about for 4.1
r
As of StarRocks v4.1, the specialized persistent row storage (enabled by 
"store_row_column" = "true"
) is still not supported for shared-data clusters. According to the latest feature compatibility matrix for StarRocks 4.1: * Row Store (Hybrid Row-Column): Remains "To be supported" for shared-data mode. This feature is currently only available for shared-nothing (local storage) clusters. * Persistent Index: This is fully supported and recommended for shared-data mode. In v4.1, you should use 
"persistent_index_type" = "CLOUD_NATIVE"
, which stores the index on S3/Object Storage while caching it locally for speed. This provides the fast primary key lookups you need for your 50ms goal. What’s New in 4.1 for Shared-Data Performance? While the dedicated row-store property is missing, v4.1 introduces other features that significantly improve performance in cloud-native environments: 1. Inverted Index Support: v4.1 brings support for inverted indexes to shared-data clusters. This allows for ultra-fast text searching and filtering on non-primary-key columns, similar to how the row store helps point lookups. 2. Range-Based Distribution: v4.1 supports range-based data distribution for both shared-nothing and shared-data clusters, which can help prune data more effectively for specific query patterns. 3. Enhanced Cloud-Native Persistent Index: Continued optimizations to the 
CLOUD_NATIVE
index type reduce the memory overhead and improve the speed of finding row locations on S3. Recommendation for your 50ms Latency To achieve your latency target in v4.1 without the row store property: 1. Warm the Data Cache: Use the 
CACHE SELECT
or warmup tasks introduced in v3.3+ to ensure your "hot" data is already on the local NVMe of your EKS nodes. 2. Cloud-Native Persistent Index: Ensure your table is created with:
Copy code
sql
    PROPERTIES (
      "enable_persistent_index" = "true",
      "persistent_index_type" = "CLOUD_NATIVE"
    );
3. Point Lookup Optimization: Since the row store is unavailable, ensure your query explicitly filters by the Primary Key to take advantage of the index. References • deployment/shared_data/feature-support-shared-data.md • table_design/table_types/primary_key_table.md • table_design/hybrid_table.md References • deployment/shared_data/feature-support-shared-data.md • table_design/table_types/primary_key_table.md • table_design/hybrid_table.md References • deployment/shared_data/feature-support-shared-data.md • knowledge/admin_and_operation/shared_data_compaction.md
k
@Rocky is there a way to somehow rewrite queries meant for an iceberg table to a Starrocks native table in shared data mode
r
Yes, you can do this transparently using Asynchronous Materialized Views (MVs) with Query Rewrite. This is the standard way to "redirect" queries from an Iceberg table to a native StarRocks table without changing your SQL or your application logic. In shared-data mode, the Materialized View acts as a native table but is backed by your cloud storage (S3). How it works 1. Create the MV: You create an asynchronous MV that selects data from your Iceberg table. This MV is technically a native StarRocks table. 2. Transparent Matching: When a user sends a query against the original Iceberg table, the StarRocks Optimizer (FE) automatically detects if the MV contains the required data. 3. Rewrite: If it matches, the FE "rewrites" the query on the fly to fetch data from the native MV instead of the Iceberg table. Why this solves your metadata issue When the rewrite happens, the FE bypasses the Iceberg catalog entirely. It uses StarRocks' internal metadata (which is in memory) to find the data tablets on S3. This eliminates the 100ms–200ms overhead of parsing Iceberg manifests and snapshots. Key Configuration To ensure the rewrite happens consistently and quickly, you should use these properties:
Copy code
sql
CREATE MATERIALIZED VIEW iceberg_acceleration_mv
DISTRIBUTED BY HASH(id)
REFRESH ASYNC EVERY (INTERVAL 10 MINUTE) -- Or MANUAL
PROPERTIES (
  -- Allows the query to be rewritten even if the MV is slightly behind the Iceberg table
  -- This is critical for sub-50ms response times.
  "mv_rewrite_staleness_second" = "3600", 
  
  -- Ensures StarRocks considers this external table for rewrite
  "force_external_table_query_rewrite" = "true"
)
AS 
SELECT id, name, timestamp FROM iceberg_catalog.db.table;
Important Considerations for Shared-Data Mode * Staleness: By default, StarRocks checks if the Iceberg table has changed. If it has, it might skip the rewrite to ensure "fresh" data, which puts you back on the slow 200ms path. By setting 
mv_rewrite_staleness_second
, you tell StarRocks: "I'm okay with data that is X seconds old; just give me the 50ms response time." * Storage: In shared-data mode, the MV data will be stored on S3, but its Persistent Index and Data Cache will live on your EKS worker nodes (BEs). * Query Pattern: The rewrite works best for "SPJG" queries (Select, Project, Join, Group-by). Simple point lookups by ID are very easy for the optimizer to rewrite. Summary: This is the most "elegant" solution because your BI tools and users keep querying 
iceberg_catalog.table
, but they get the performance of a native StarRocks table. References • using_starrocks/async_mv/use_cases/data_lake_query_acceleration_with_materialized_views.md • using_starrocks/async_mv/use_cases/query_rewrite_with_materialized_views.md • sql-reference/sql-statements/materialized_view/CREATE_MATERIALIZED_VIEW.md References • using_starrocks/async_mv/use_cases/data_lake_query_acceleration_with_materialized_views.md • using_starrocks/async_mv/use_cases/query_rewrite_with_materialized_views.md • sql-reference/sql-statements/materialized_view/CREATE_MATERIALIZED_VIEW.md References
k
@Rocky is there a way to somehow rewrite queries meant for an iceberg table to a Starrocks native table in shared data mode without materialized views
r
Without using Materialized Views (MVs), there is no feature in StarRocks that provides transparent query rewrite (where StarRocks automatically "hijacks" a query specifically targeting 
iceberg_catalog.db.table
and redirects it to an internal table). Transparent rewrite is a core logic of the MV engine. However, you can achieve a similar result using Logical Views and Catalog Management, provided you can make a small configuration change at the session or application connection level. The "Catalog Shadowing" Strategy If you cannot use MVs but want to "redirect" queries to a native StarRocks table in shared-data mode, you can use the following approach: 1. Create a Native Table: Create your StarRocks native table in the 
default_catalog
.
Copy code
sql
    CREATE TABLE default_catalog.my_db.native_iceberg_data (...)
    PROPERTIES ("enable_persistent_index" = "true");
2. Create a Logical View: Create a view in the 
default_catalog
that has the exact same name as your Iceberg table.
Copy code
sql
    CREATE VIEW default_catalog.my_db.iceberg_table_name AS
    SELECT * FROM default_catalog.my_db.native_iceberg_data;
3. Switch the Session Catalog: Configure your application or BI tool to run 
SET CATALOG default_catalog;
and 
USE my_db;
upon connecting. How it works: If the application sends the query 
SELECT _ FROM iceberg_table_name
, StarRocks will look in the _current* catalog and database first. It will find your Logical View (which points to the native table) and execute it, completely bypassing the actual Iceberg catalog. Why this is different from MVs * No Fully Qualified Names: This only works if your application does not use fully qualified names (e.g., it must query 
table_name
instead of 
iceberg_catalog.db.table_name
). If the application explicitly prefixes the Iceberg catalog, StarRocks will always go to Iceberg. * Manual Sync: Unlike an MV, which has a 
REFRESH
property, you are responsible for manually keeping the data in your native table synchronized with the source Iceberg table (e.g., using 
INSERT INTO ... SELECT
). * No "Staleness" Logic: A Logical View is just a pointer. It doesn't have the sophisticated consistency checks or "staleness" tolerances that MVs provide. Summary: Can you do it without MVs? Feature | Asynchronous MV | Logical View + Catalog Switch Transparent? | Yes (handles 
iceberg_catalog
prefix) | No (requires unqualified names) Metadata Speed | Fast (Internal) | Fast (Internal) Data Freshness | Automatic (Refresh) | Manual (Insert into) Rewrite Logic | Built-in Optimizer | Name Matching Recommendation: If you are trying to avoid MVs because of the "background refresh" overhead, consider using an MV with 
REFRESH MANUAL
. This gives you the Transparent Rewrite (so you don't have to change your SQL) but lets you control exactly when data moves, just like a native table. References • data_source/catalog/catalog_overview.md • sql-reference/sql-statements/View/CREATE_VIEW.md • using_starrocks/async_mv/use_cases/query_rewrite_with_materialized_views.md References • data_source/catalog/catalog_overview.md • sql-reference/sql-statements/View/CREATE_VIEW.md • using_starrocks/async_mv/use_cases/query_rewrite_with_materialized_views.md • data_source/catalog/default_catalog.md References
k
@Rocky Does shared data mode support primary key tables?
r
Yes, shared-data mode fully supports Primary Key tables. The support for Primary Key tables in shared-data clusters was introduced in StarRocks v3.1. Since then, several key features have been added to make them performant in cloud-native environments: Key Features and Version Support * Primary Key Tables: Supported since v3.1.0. * Partial Updates: Supported in Row mode since v3.1.0 and in Column mode since v3.3.1. * Persistent Index (Local): Supported since v3.1.4. This stores the index on the local disk of the BE/CN nodes to save memory. * Cloud-Native Persistent Index: Supported since v3.3.2. This allows the index to be stored on object storage (like S3) while being cached locally. This is the recommended configuration for v4.1 as it allows for much larger datasets without being limited by local disk or memory. Recommended Configuration for v4.1 To get the best performance and scalability for a Primary Key table in shared-data mode, you should enable the cloud-native persistent index:
Copy code
sql
CREATE TABLE my_table (
    id BIGINT NOT NULL,
    name VARCHAR(65533),
    ...
) PRIMARY KEY (id)
DISTRIBUTED BY HASH(id)
PROPERTIES (
    "enable_persistent_index" = "true",
    "persistent_index_type" = "CLOUD_NATIVE" -- Recommended for shared-data
);
Limitations to Keep in Mind * Hybrid Row-Column Storage: As mentioned previously, the specialized row-store property (
"store_row_column" = "true"
) is not yet supported in shared-data mode, even in v4.1. * Index Warmup: Because data and indexes are stored on S3, the first few queries after a node restart or cache eviction may be slower while the index is being pulled from object storage into the local cache. References • deployment/shared_data/feature-support-shared-data.md • table_design/table_types/primary_key_table.md • sql-reference/sql-statements/table_bucket_part_index/CREATE_TABLE.md References • deployment/shared_data/feature-support-shared-data.md • table_design/table_types/primary_key_table.md
2 Views
Open in Slack
PreviousNext
Powered by