@Rocky I'm trying to create a very expensive materialized view on my shared nothing cluster. It's referencing an iceberg catalog. The theory is that materialized view will be stored on disk in the shared nothing cluster vs relying it being cached in the NVMe in the shared data and possibly requiring s3 fetches. My BE nodes always spike in CPU (but not memory) and eventually restarts as CPU approaches 100%. What is the issue here? I thought high cpu usage means it just gets throttled? Memory is nowhere near 100%, no not OOMing

Creating a large Materialized View (MV) on an Iceberg catalog involves intensive metadata parsing (manifest files) and data scanning, which can easily pin CPU resources. Since your memory remains stable but the BE process restarts as CPU approaches 100%, the issue is likely thread starvation leading to a health check failure. Root Cause: Health Check/Liveness Probe Failure When the BE CPU usage hits 100%, the process may become unresponsive to external "heartbeats" or health checks. * Kubernetes: If you are running on K8s, the

livenessProbe

often has a short timeout (e.g., 1–5 seconds). When the BE is pinned at 100% CPU, it may fail to respond to the probe in time, causing K8s to kill and restart the container. * StarRocks Heartbeat: The FE sends heartbeats to the BE every 5 seconds. If the BE is so busy that the heartbeat thread is starved for 3 consecutive intervals (default 15s), the FE will mark the BE as "Dead." While this shouldn't kill the process, it stops task execution and can lead to cascading failures. * Systemd Watchdog: In some on-premise deployments, a systemd watchdog may be configured to restart the service if it doesn't "notify" within a specific window. Recommended Solutions 1. Limit MV Resource Consumption (Resource Groups) By default, a heavy query or MV creation will attempt to use all available CPU cores. You should constrain the MV refresh task so it leaves room for system threads and heartbeats. * Check/Set MV CPU Limits: StarRocks has a built-in resource group for MVs called

default_mv_wg

. You can limit its CPU usage (per BE) via `be.conf`:

properties
    # Limit the number of CPU cores the MV refresh can use per BE
    # Default is often 1 or 0 (unlimited) depending on version.
    # Set it to roughly 50-70% of your total cores.
    default_mv_resource_group_cpu_limit = 8

* Using Resource Groups: You can also create a custom resource group for your MV user/query and set

cpu_core_limit

. 2. Adjust Health Check Timeouts If you are on K8s, increase the

livenessProbe

and

readinessProbe

thresholds to be more "forgiving" during heavy loads: * `failureThreshold`: Increase from 3 to 10 or more. * `timeoutSeconds`: Increase to 5 or 10 seconds. * `initialDelaySeconds`: Ensure the BE has enough time to fully initialize. 3. Optimize Iceberg Scanning Since you are pulling from Iceberg to store locally, the initial population is essentially a massive

INSERT INTO ... SELECT

. * Partitioning: If the Iceberg table is large, create the MV with a partition strategy (

PARTITION BY

). StarRocks can then refresh the data partition-by-partition rather than one massive global task. * Parallelism: If CPU is the bottleneck, you can try reducing the scan parallelism for that specific task:

sql
    SET TABLE odf_iceberg_table PROPERTIES ("connector_io_tasks_per_scan_operator" = "2");

4. Verify the "Restart" Reason To confirm if it's a crash vs. an external kill: * Check

be.out

and

<http://be.INFO|be.INFO>

logs: Look for "Internal Error," "Panic," or "Segmentation Fault." If the log simply ends and starts over, it was likely killed by K8s or the OS. * K8s Events: Run

kubectl describe pod <be-pod-name>

and look for "Liveness probe failed" or "OOMKilled" (though you noted memory is fine, "Terminated" with exit code 137 usually means SIGKILL). Summary Checklist 1. Throttling: Set

default_mv_resource_group_cpu_limit

in

be.conf

to prevent the MV from taking 100% of the node. 2. K8s Probes: Relax liveness probe settings. 3. Partitioning: Ensure the MV is partitioned to avoid one giant, non-preemptible CPU spike of-preemptible CPU spike. References • administration/management/BE_configuration.md • administration/management/monitoring/alert.md

@Rocky

FE should be able to route this automatically. Not something we have to handle on the client side.

[1:52 PM] Just give it the fe dns and it should handle it automatically.

Is this true? I have a periodic leader IP refresh client side so that my stream load always targets the leader node. Will FE automatically handle routing correctly? So I can just send requests to the round robin endpoint that distributes traffic to 3 of my FE nodes?

Yes, it is true. You do not need to manually refresh the leader FE IP on the client side for Stream Load. You can simply point your client (or a round-robin load balancer) to any of your FE nodes. How it works StarRocks FEs are designed to handle requests in a high-availability (HA) manner: 1. Internal Forwarding: If you send a Stream Load request to a Follower or Observer FE, that node will internally forward the transaction metadata (like "beginning" and "committing" the load) to the current Leader FE. 2. HTTP 307 Redirect: Once the FE has coordinated with the Leader, it selects a BE to act as the "Coordinator" for that load and returns an HTTP 307 (Temporary Redirect) to your client. 3. Data Flow: Your client (if using

--location-trusted

with curl or a standard HTTP client that follows redirects) then sends the actual data directly to that BE. This means the high-bandwidth data transfer never saturates the FE, and it doesn't matter which FE you hit first. Recommendations * Use an LB: In production, it is best practice to put a Load Balancer (like HAProxy, Nginx, or a Cloud LB) in front of all your FEs. This ensures that if one FE goes down, your loading pipeline remains uninterrupted. * Client Configuration: Ensure your loading tool follows redirects. For example, with

curl

, you must use the

-L

(or

--location

) and

--location-trusted

flags to ensure credentials are passed to the redirected BE address. * Direct to BE (Optional): You can send requests directly to a BE, but it is generally recommended to go through an FE because the FE will perform load balancing across all available BEs for you. Summary: Your client-side leader refresh logic is redundant. You can safely switch to a round-robin endpoint or a fixed list of FE DNS names/IPs. References • loading/StreamLoad.md • sql-reference/sql-statements/loading_unloading/STREAM_LOAD.md • administration/management/resource_management/Load_balance.md • quick_start/helm.md • loading/Stream_Load_transaction_interface.md References • loading/StreamLoad.md • quick_start/helm.md • administration/management/FE_configuration.md • quick_start/shared-data.md • administration/management/BE_configuration.md • sql-reference/sql-statements/loading_unloading/STREAM_LOAD.md