http://starrocks.io logo
Join Slack
Powered by
<@U0A71G31CDV> - Show me example to create an incr...
# questions-and-troubleshooting
a
@Rocky - Show me example to create an incremental refresh materialized view from iceberg catalog. Need it as close to real-time, as possible.
r
To create an incremental refresh materialized view from an Iceberg catalog that stays as close to real-time as possible, you should use the 
INCREMENTAL
 refresh mode combined with a high-frequency 
ASYNC
 refresh interval. Starting with StarRocks v3.3 and refined in v4.x, the 
INCREMENTAL
mode (Incremental View Maintenance) allows StarRocks to scan only the new data (deltas) in the Iceberg table since the last snapshot, rather than recomputing entire partitions. SQL Example
Copy code
sql
CREATE MATERIALIZED VIEW iceberg_incremental_mv
-- 1. Partition the MV to match the base table for efficiency
PARTITION BY (dt)
DISTRIBUTED BY HASH(user_id)
-- 2. Set the refresh interval to 1 minute for near real-time updates
REFRESH ASYNC EVERY (INTERVAL 1 MINUTE)
PROPERTIES (
    -- 3. Use INCREMENTAL mode to process only deltas (Iceberg snapshots)
    "refresh_mode" = "INCREMENTAL"
)
AS 
SELECT 
    user_id,
    dt,
    count(event_id) as event_count,
    sum(amount) as total_amount
FROM iceberg_catalog.sales_db.orders
GROUP BY user_id, dt;
Key Components for "Near Real-Time" Performance 1. 
refresh_mode = INCREMENTAL
: This is the most critical property. Instead of performing a "Partition-level refresh" (which re-scans entire partitions where data changed), StarRocks uses Iceberg's snapshot metadata to identify and process only the newly appended rows. 2. 
REFRESH ASYNC EVERY (INTERVAL 1 MINUTE)
: This is the shortest recommended interval for asynchronous refreshes. It ensures the MV task is triggered frequently. 3. Partition Alignment: If the base Iceberg table is partitioned (e.g., by 
dt
), partitioning the Materialized View by the same column allows StarRocks to manage data more efficiently and support partition-level TTL if needed. 4. Append-Only Requirement: The 
INCREMENTAL
refresh mode currently supports append-only operations on the base Iceberg table. If the source table undergoes 
UPDATE
, 
DELETE
, or 
OVERWRITE
operations: * The refresh will fail if 
refresh_mode
is set to 
INCREMENTAL
. * The system will fall back to a full/partition-level refresh if 
refresh_mode
is set to 
AUTO
. How to Check Refresh Status You can monitor if the MV is truly refreshing incrementally by checking the 
information_schema.task_runs
table:
Copy code
sql
SELECT 
    task_name, 
    create_time, 
    finish_time, 
    state, 
    get_json_string(extra_message, '$.refreshMode') as refresh_mode
FROM information_schema.task_runs 
ORDER BY create_time DESC LIMIT 5;
Important Considerations * Iceberg Version: Incremental detection is most robust for Iceberg V1 tables. For Iceberg V2 tables with equality/position deletes, ensure you are on StarRocks v3.3+ where support for these formats is being actively expanded. * Operator Support: Incremental refresh supports standard operators like 
SELECT
, 
FILTER
, 
INNER JOIN
, and 
AGGREGATE
(with 
GROUP BY
). It does not yet support 
DISTINCT
or 
OUTER JOIN
in true incremental mode. References • sql-reference/sql-statements/materialized_view/CREATE_MATERIALIZED_VIEW.md • using_starrocks/async_mv/use_cases/data_lake_query_acceleration_with_materialized_views.md
a
@Rocky - starrocks 4.1.1 shared-data deployment on k8s - 4 fe nodes, 3 cn nodes
r
For a StarRocks 4.1.1 shared-data (compute-storage separation) deployment on Kubernetes, you should use the StarRocks Operator. In shared-data mode, the architecture replaces traditional Backends (BE) with Compute Nodes (CN). Data is stored in object storage (e.g., S3, MinIO, GCS), while CNs handle the computation and local caching. 1. Topology Recommendation While you requested 4 FE nodes, please note that StarRocks FEs use the Raft protocol for metadata consensus, which requires a quorum (majority). * 3 FEs: Can survive 1 failure (Quorum = 2). * 4 FEs: Can still only survive 1 failure (Quorum = 3). * Recommendation: Use 3 or 5 FE nodes for optimal fault tolerance. 2. Configuration Requirements To enable shared-data mode, you must configure the following in both 
fe.conf
and `cn.conf`: * 
run_mode = shared_data
* Object storage credentials (S3 endpoint, bucket, access keys). 3. StarRocksCluster YAML Example Below is an example manifest for the StarRocks Operator. This assumes you are using S3-compatible storage (like AWS S3 or MinIO).
Copy code
yaml
apiVersion: <http://starrocks.com/v17|starrocks.com/v17>  # Check your operator version for the correct API
kind: StarRocksCluster
metadata:
  name: starrocks-shared-data
  namespace: starrocks
spec:
  starRocksFeSpec:
    replicas: 3  # Recommended over 4 for quorum efficiency
    image: starrocks/fe-ubuntu:4.1.1
    configMapInfo:
      configMapName: fe-config
      resolveKey: fe.conf
    storageVolumes:
      - name: fe-meta
        storageClassName: standard
        storageSize: 10Gi
        mountPath: /opt/starrocks/fe/meta

  starRocksCnSpec:
    replicas: 3
    image: starrocks/cn-ubuntu:4.1.1
    configMapInfo:
      configMapName: cn-config
      resolveKey: cn.conf
    storageVolumes:
      - name: cn-cache
        storageClassName: local-path # High-performance SSD recommended for cache
        storageSize: 100Gi
        mountPath: /opt/starrocks/cn/storage
4. ConfigMap Details You must create the ConfigMaps referenced above before applying the cluster manifest. FE Config (
fe.conf
):
Copy code
properties
run_mode = shared_data
cloud_native_storage_type = S3
aws_s3_path = my-bucket/starrocks_metadata
aws_s3_endpoint = <http://s3.us-east-1.amazonaws.com|s3.us-east-1.amazonaws.com>
aws_s3_region = us-east-1
aws_s3_access_key = <YOUR_ACCESS_KEY>
aws_s3_secret_key = <YOUR_SECRET_KEY>
# Enable storage volume access check
enable_storage_volume_access_check = true
CN Config (
cn.conf
):
Copy code
properties
# CN is shared_data by default, but explicitly setting it is safer
sys_log_level = INFO
# Ensure the cache path matches your storageVolume mountPath
storage_root_path = /opt/starrocks/cn/storage
5. Deployment Steps 1. Install Operator: Ensure the StarRocks Operator is installed in your K8s cluster. 2. Create Secrets/ConfigMaps: Apply the 
fe-config
and 
cn-config
ConfigMaps containing your S3 credentials. 3. Apply Cluster: Run 
kubectl apply -f starrocks-cluster.yaml
. 4. Verify: * Check pods: 
kubectl get pods -n starrocks
* Check nodes in MySQL: 
SHOW FRONTENDS;
and 
SHOW COMPUTE NODES;
Important for v4.1.1 StarRocks 4.1.x continues to enhance the Warehouse concept. In shared-data mode, you can group your 3 CN nodes into a specific warehouse for workload isolation:
Copy code
sql
CREATE WAREHOUSE compute_wh WITH PROPERTIES ("type" = "compute");
References • deployment/sr_operator.md • deployment/shared_data/s3.md • quick_start/shared-data.md References • deployment/shared_data/feature-support-shared-data.md • deployment/sr_operator.md • deployment/shared_data/s3.md
Open in Slack
PreviousNext
Powered by