http://starrocks.io logo
Join Slack
Powered by
<@U0A71G31CDV> in 4.1.4 shared nothing , if a new ...
# questions-and-troubleshooting
k
@Rocky in 4.1.4 shared nothing , if a new backend node is added to the cluster, how does that impact existing tables and materialized views?
r
In StarRocks v4.1.4 (shared-nothing architecture), adding a new Backend (BE) node impacts existing tables and materialized views through automated background rebalancing and the new Adaptive Layout capabilities introduced in the 4.1 release line. 1. Automatic Tablet Rebalancing When you add a new BE node using 
ALTER SYSTEM ADD BACKEND
, the system's Tablet Scheduler immediately detects the new node's low load. * Load Balancing: StarRocks calculates a 
loadScore
for every BE based on disk utilization and replica count. It then begins moving tablets from high-load nodes to the new node. * Safety: The process is online and non-disruptive. For each tablet being moved, StarRocks first creates a new replica on the new BE. Only after the new replica is synchronized does it drop the redundant replica on the original node. * Impact: Queries against existing tables and materialized views continue to run during this process. Since rebalancing is a background task, it is designed to have a minimal impact on cluster performance. 2. Adaptive Layout (New in v4.1) One of the most significant changes in version 4.1 is the Adaptive Layout feature. * Dynamic Adjustment: Unlike earlier versions where tablet count (buckets) was a one-time decision made at table creation, v4.1 allows the system to observe data distribution and adjust the physical layout at runtime. * Automatic Splitting: If the addition of the new BE makes the existing tablet count suboptimal for the new cluster size, the system can automatically split tablets to achieve better parallelism and distribution across the expanded resources. This ensures that a table with a small number of buckets can still scale effectively across a larger number of BE nodes. 3. Impact on Materialized Views Materialized views (MVs) are treated similarly to physical tables because they store pre-computed results in their own tablets. * Rebalancing: The tablets belonging to an MV will be rebalanced across the new BE node just like regular table tablets. * Refresh Behavior: The refresh process of an asynchronous MV is not inherently affected by the addition of a BE node, other than benefiting from the increased compute and storage bandwidth of the larger cluster. * Data Distribution Labels: If you have configured specific BE labels for your MVs (e.g., 
labels.location
), you may need to update the MV's properties or add the same labels to the new BE node to ensure the MV's data can be distributed to it. Summary of Actions * Command: 
ALTER SYSTEM ADD BACKEND "be_host:heartbeat_service_port";
* Monitoring: You can monitor the progress of the rebalancing using:
Copy code
sql
    SHOW PROC '/cluster_balance/pending_tablets';
    SHOW PROC '/cluster_balance/running_tablets';
* No Manual Intervention: For standard tables and MVs, no manual data migration or MV rebuild is required. The system handles the transition automatically. References • administration/management/resource_management/be_label.md • faq/operation_maintenance_faq.md • administration/management/FE_configuration.md • using_starrocks/async_mv/use_cases/data_modeling_with_materialized_views.md • using_starrocks/async_mv/use_cases/query_rewrite_with_materialized_views.md
k
@Rocky Refresh mv my_mv failed after 3 times, try lock failed: 0, error-msg : com.starrocks.common.DdlException: load channel: 01a0a3ca-d587-7f97-b1e2-2454ef042f15 was aborted at 1789456997, reason: Cancelled: Cancelled by pipeline engine, reason: Memory limit exceeded: Memory of process exceed limit. try consume:8388608 Backend: starrocks-etl-cn-2.starrocks-etl-cn-search.starrocks-etl.svc.cluster.local, Used: 48712363504, Limit: 50444390890. Mem usage has exceed the limit of BE: BE10007 be10.0.70.45 BE:10004 at com.starrocks.common.ErrorReport.reportDdlException(ErrorReport.java:100) at com.starrocks.qe.StmtExecutor.handleDMLStmt(StmtExecutor.java:3442) [wrapped] com.starrocks.common.StarRocksException: load channel: 01a0a3ca-d587-7f97-b1e2-2454ef042f15 was aborted at 1789456997, reason: Cancelled: Cancelled by pipeline engine, reason: Memory limit exceeded: Memory of process exceed limit. try consume:8388608 Backend: starrocks-etl-cn-2.starrocks-etl-cn-search.starrocks-etl.svc.cluster.local, Used: 48712363504, Limit: 50444390890. Mem usage has exceed the limit of BE: BE10007 be10.0.70.45 BE:10004 why is this not spilling to disk if i have the variable set global enable_spill = true;
r
Setting 
SET GLOBAL enable_spill = true;
does not automatically enable spilling for asynchronous materialized view (MV) refresh tasks. In StarRocks, background MV refresh tasks use session variables defined specifically in the MV's properties. There are two main reasons why your MV is still hitting memory limits: 1. Spilling is not enabled in the MV properties Asynchronous MVs do not inherit all global session variables for their background refresh jobs. To enable spilling for an existing MV, you must explicitly set the property with the 
session.
prefix:
Copy code
sql
ALTER MATERIALIZED VIEW my_mv SET ('session.enable_spill' = 'true');
You can verify if this is set by running 
SHOW CREATE MATERIALIZED VIEW my_mv;
and looking for the properties section. 2. Spilling does not cover the "Load Channel" Your error message specifically mentions 
load channel: ... was aborted
. * What spills: Only intermediate results from Aggregate, Sort, and Join operators can spill to disk. * What does NOT spill: The Load Channel (the "Sink" phase where data is buffered before being written to tablets) does not support spilling. If the memory is being exhausted during the write phase (the "Load Channel"), spilling will not prevent the OOM. This happens most often if: * Too many buckets/tablets: If the MV has a very large number of buckets, each bucket requires a memory buffer in the load channel. * Wide rows: Very wide tables increase the buffer size per bucket. * High concurrency: Other load jobs or queries are consuming the BE's process memory simultaneously. Recommended Solutions 1. Enable Spill via Property: Apply the 
ALTER
statement above to ensure the computation part of the refresh (Agg/Join) spills if needed. 2. Use Partitioned Refresh: Instead of refreshing the entire MV, use a 
PARTITION BY
clause in the MV definition. This allows StarRocks to refresh only the changed partitions, significantly reducing the memory footprint per refresh task. 3. Check Bucket Count: Verify the bucket count for your MV. If it's unnecessarily high (e.g., hundreds or thousands of tablets per BE), reduce it to lower the load channel memory consumption. 4. Tune Write Buffers (BE Config): You can try reducing the 
write_buffer_size
in 
be.conf
(default is 100MB) to reduce memory usage per tablet writer, though this may impact write performance. 5. Increase Memory Limit: If the BE has physical RAM available, ensure 
mem_limit
in 
be.conf
is set appropriately (usually 90%). The error shows you are hitting the process limit (
48.7GB / 50.4GB
). To diagnose further: Trigger a manual refresh and check the query profile to see which operator is consuming the most memory:
Copy code
sql
-- Enable profile collection for the MV
ALTER MATERIALIZED VIEW my_mv SET ('session.enable_profile' = 'true');
-- Manually trigger refresh
REFRESH MATERIALIZED VIEW my_mv;
-- View the profile in the FE web UI or via:
get_query_profile('<query_id>');
References • administration/management/FE_configuration.md • using_starrocks/async_mv/troubleshooting_asynchronous_materialized_views.md • administration/management/resource_management/spill_to_disk.md • loading/loading_introduction/loading_considerations.md References • administration/management/FE_configuration.md • using_starrocks/async_mv/troubleshooting_asynchronous_materialized_views.md
k
@Rocky MV rewrite fail for my_mv: stale partitions refreshType=FULL why does this happen if the MV is not partitioned? and the staleness second expires beyond when this query ran?
r
This error occurs because of how StarRocks internally tracks data consistency for Materialized Views (MVs), even when they are not partitioned. 1. "Stale Partitions" in Non-Partitioned MVs In StarRocks, every table and MV is internally treated as having at least one partition (the "default" partition). When you see 
stale partitions refreshType=FULL
for a non-partitioned MV, it simply means the system has detected that the base table data has changed since the last successful refresh, marking the MV's single internal partition as "stale." 2. Why it fails despite 
staleness_seconds
 If you have set 
mv_rewrite_staleness_second
but the rewrite still fails with this error, it is usually due to one of the following reasons: * No Successful Initial Refresh: The 
mv_rewrite_staleness_second
property relies on the timestamp of the last successful refresh. If your MV refresh has been failing (e.g., due to the OOM error you shared earlier) and has never completed successfully, there is no "last refresh time" to compare against. In this state, the MV is considered completely empty or invalid for rewrite. * Property Name Typo: Ensure you are using the exact property name: 
mv_rewrite_staleness_second
(singular). If you used 
staleness_seconds
or a similar variation, the property will be ignored. _ _Correct usage:* 
ALTER MATERIALIZED VIEW my_mv SET ("mv_rewrite_staleness_second" = "3600");
* Inactive MV State: If a base table's schema was modified, StarRocks marks the MV as 
inactive
. When an MV is inactive, it cannot be used for query rewrite, regardless of the staleness settings. _ _Check state:* 
SHOW MATERIALIZED VIEWS LIKE 'my_mv';
(Check the 
is_active
column). _ _Fix:* 
ALTER MATERIALIZED VIEW my_mv ACTIVE;
* Consistency Mode: If the session variable 
query_rewrite_consistency
is explicitly set to 
checked
, the optimizer may be more strict. Try setting it to 
loose
at the session level to see if the rewrite triggers:
Copy code
sql
    SET query_rewrite_consistency = 'loose';
How to verify To see the exact reason the optimizer rejected the MV, use the 
TRACE
command:
Copy code
sql
TRACE REASON MV <your_query_here>;
This will provide a more granular explanation of why the "stale partitions" check failed or why the staleness property wasn't applied. Recommendation: Since your refresh is failing with OOM, focus on getting at least one successful refresh completed (by increasing memory or enabling spilling in the MV properties as discussed). Once a valid snapshot exists, your staleness window will begin to work. References • using_starrocks/async_mv/troubleshooting_asynchronous_materialized_views.md • using_starrocks/async_mv/use_cases/create_partitioned_materialized_view.md
k
@Rocky what is the meaning of this failure, other iceberg queries are fine: reload failed: Failed to init connector [type: iceberg, name: iceberg]. msg: null
r
This error message indicates that StarRocks attempted to refresh or re-initialize the Iceberg catalog (a process called "reload"), but the operation failed because the initialization logic encountered an unhandled exception—likely a 
NullPointerException
(hence 
msg: null
). The reason your other queries are still working is that StarRocks uses a "graceful reload" strategy: it tries to create a new instance of the connector in the background. If the new one fails to initialize, it keeps the old, working connector active so that user queries aren't interrupted. Common Causes for this failure: 1. Expired or Missing Credentials: If you are using an Iceberg REST catalog with OAuth2 or an AWS Glue catalog with temporary tokens, the background thread might be failing to refresh the credentials. If the refresh logic fails, the "reload" of the connector will fail. (A known issue similar to this was addressed in StarRocks v4.1.2). 2. Missing Mandatory Properties: If the catalog properties were recently modified (via 
ALTER CATALOG
), the new configuration might be missing a field that the Iceberg client requires. Even a typo in a property name can cause the internal 
init
method to fail with a null error when it tries to access a missing value. 3. Metastore Connectivity/Response Issues: If you are using a REST Catalog, the error can occur if the REST server returns an unexpected or empty configuration response. If using Hive Metastore (HMS), it could be a transient failure to load the Hadoop/Hive configuration files during the reload attempt. 4. Metadata Inconsistency: If a specific table in the Iceberg catalog has corrupted metadata (e.g., a missing 
metadata-log
entry or an invalid snapshot), the StarRocks metadata crawler might hit an exception when trying to list or initialize the tables during the reload. How to Troubleshoot: * Check the FE Logs (Most Important): Search the 
fe.log
or 
fe.warn.log
on your leader FE node for the string 
Failed to init connector
. Because the error message in the SQL client is 
msg: null
, you need the Java stack trace from the log file to see exactly which line of code triggered the error. * Verify Catalog Properties: Run 
SHOW CREATE CATALOG <catalog_name>;
and ensure all required fields (like 
iceberg.catalog.uri
, 
iceberg.catalog.type
, and credentials) are present and correct. * Manual Refresh: Try to force a refresh manually and see if it provides a better error:
Copy code
sql
    REFRESH EXTERNAL CATALOG iceberg;
* Check for Token Expiry: If using a REST catalog, ensure the 
iceberg.catalog.oauth2.credential
or 
token
has not expired. Note: If you are on a version earlier than 4.1.2, this might be related to a known bug (#76457) where REST catalog tokens weren't being refreshed correctly. Upgrading to a later maintenance release is recommended. References • data_source/catalog/iceberg/iceberg_catalog.md
k
@Rocky times, try lock failed: 0, error-msg : com.starrocks.common.StarRocksException: Failed to allocate resource to query: pending timeout [300s], you could modify the session variable [query_queue_pending_timeout_second] to pending more time at com.starrocks.qe.QueryQueueManager.maybeWait(QueryQueueManager.java:106) at com.starrocks.qe.DefaultCoordinator.startScheduling(DefaultCoordinator.java:568) at com.starrocks.qe.scheduler.Coordinator.exec(Coordinator.java:96) at com.starrocks.qe.StmtExecutor.handleDMLStmt(StmtExecutor.java:3393) [wrapped] com.starrocks.common.StarRocksException: Failed to allocate resource to query: pending timeout [300s], you could modify the session variable [query_queue_pending_timeout_second] to pending more time at
r
The error 
pending timeout [300s]
indicates that your Materialized View (MV) refresh task is being held in the Query Queue and timed out before it could start. StarRocks uses a query queue to protect the cluster from crashing when resources (CPU or Memory) are exhausted. Since your previous error showed the BE memory was at 96.6% usage (48.7GB/50.4GB), the system is intentionally blocking new heavy tasks like MV refreshes to prevent an Out-Of-Memory (OOM) crash. Why this is happening 1. Memory Threshold: By default, if BE memory usage exceeds 90%, new queries are placed in a 
PENDING
state in the queue. 2. Concurrency Limits: If there are already many concurrent queries or refreshes running, the task must wait for a "slot." 3. Timeout: The task waited for the default 300 seconds (
query_queue_pending_timeout_second
) but the cluster resources never freed up enough to let it run. How to resolve it 1. Increase the Queue Timeout If you want the MV to wait longer instead of failing immediately, increase the global timeout:
Copy code
sql
SET GLOBAL query_queue_pending_timeout_second = 900; -- Increase to 15 minutes
2. Check what is blocking the queue Run this command to see if the refresh task is stuck and what else is running:
Copy code
sql
SHOW RUNNING QUERIES;
Look for 
State = PENDING
. If many queries are pending, your cluster is over-capacity. 3. Adjust MV Resource Group Limits Asynchronous MVs use a hidden resource group called 
default_mv_wg
. You can increase its allowed concurrency or memory threshold via FE configurations (requires admin access): * 
default_mv_resource_group_memory_limit
(Default 80%) * 
default_mv_resource_group_concurrency_limit
(Default 0, meaning no specific limit other than global) Alternatively, assign the MV to a specific resource group with higher priority:
Copy code
sql
ALTER MATERIALIZED VIEW my_mv SET ("resource_group" = "your_high_priority_rg");
4. Address the Root Cause (Memory) Since your earlier logs showed the BE is nearly out of memory, the queue is actually doing its job. To stop the queueing, you must lower memory pressure: * Enable Spilling: (As mentioned previously) Ensure 
session.enable_spill
is 'true' in the MV properties so the refresh uses disk instead of RAM. * Decrease Concurrency: Reduce the number of simultaneous MVs refreshing at the same time. * Partition the MV: If the MV is not partitioned, a refresh is a "Full" refresh which is very memory-intensive. Partitioning allows StarRocks to refresh only small chunks of data at a time. Summary of relevant variables Variable | Default | Description 
query_queue_pending_timeout_second
| 300s | How long a task waits in queue before failing. 
query_queue_concurrency_limit
| 0 (off) | Global limit on concurrent queries. 
query_queue_mem_used_pct_limit
| 0.9 (90%) | Memory usage % at which queueing starts. References • administration/management/resource_management/query_queues.md • using_starrocks/async_mv/troubleshooting_asynchronous_materialized_views.md • sql-reference/System_variable.md References • administration/management/resource_management/query_queues.md • faq/resource_isolation_faq.md • faq/Sql_faq.md • administration/management/monitoring/alert.md
k
@Rocky why is query rewirte suddently not working for anything? the mv staleness is valid and i can query the mv directly
r
The fact that query rewrite has stopped working—even though the MV is queryable and staleness is technically valid—is almost certainly tied to the Iceberg connector reload failure you encountered earlier. Here is the chain of events causing this: 1. Metadata Verification Failure (The Root Cause) To perform a transparent rewrite, StarRocks must verify that the base table data has not changed beyond your 
mv_rewrite_staleness_second
limit. * For Iceberg, StarRocks checks the 
snapshot_id
of the base table. * The error 
Failed to init connector [type: iceberg]
means the background metadata crawler is failing. * Because the connector cannot initialize, the optimizer cannot fetch the current snapshot ID from the Iceberg catalog. * If StarRocks cannot confirm the base table's version, it must assume the data is potentially inconsistent and will fail the rewrite to ensure query correctness. This results in the 
stale partitions
message you saw earlier. 2. The "Staleness" Paradox Even if you set a high 
mv_rewrite_staleness_second
, the system still needs to reach the catalog to determine when the last change happened or what the current version is. If the connection to the Iceberg catalog is broken (the "init connector" error), the staleness logic cannot execute, and the rewrite defaults to "fail safe" (no rewrite). 3. Materialized View "Inactive" State Check if the MV has been automatically deactivated due to the refresh failures or the catalog errors:
Copy code
sql
SHOW MATERIALIZED VIEWS LIKE 'my_mv';
Look at the 
is_active
column. If it is 
false
, the MV will be ignored for all rewrites regardless of staleness. You can try to force it back:
Copy code
sql
ALTER MATERIALIZED VIEW my_mv ACTIVE;
4. Resource Queue Impact The 
pending timeout
error you saw earlier means the FE's internal task manager is overwhelmed or thinks the cluster is too low on memory to even plan the refresh. If a refresh task is stuck in the queue or failed to complete even once, the MV won't have the necessary "last success" metadata required to trigger a rewrite. Recommended Steps: 1. Fix the Iceberg Catalog: The 
msg: null
error during reload is a blocker. If you are using an Iceberg REST or Glue catalog, check your FE logs (
fe.log
) for the full Java stack trace. It often points to an expired token or a networking issue that "Graceful Reload" is struggling with. 2. Restart the Connector: If possible, try dropping and recreating the Iceberg catalog (if it doesn't affect too many objects) or manually running 
REFRESH EXTERNAL CATALOG <name>
. 3. Check Rewrite Reason: Run this command to get the definitive reason from the optimizer:
Copy code
sql
    TRACE REASON MV <your_query_here>;
If it says "Base table has changed" or "Cannot get metadata," it confirms the connector issue is the culprit. 4. Loose Consistency (Workaround): To bypass the check temporarily while you fix the catalog, set the consistency to loose in your session:
Copy code
sql
    SET query_rewrite_consistency = 'loose';
References • using_starrocks/async_mv/troubleshooting_asynchronous_materialized_views.md • using_starrocks/async_mv/use_cases/data_lake_query_acceleration_with_materialized_views.md • data_source/catalog/iceberg/iceberg_catalog.md References • data_source/catalog/iceberg/iceberg_catalog.md • using_starrocks/async_mv/use_cases/query_rewrite_with_materialized_views.md
k
@Rocky this is my properties REFRESH SCHEDULE EVERY(INTERVAL 4 HOUR) PROPERTIES ( "replicated_storage" = "true", "replication_num" = "3", "mv_rewrite_staleness_second" = "259200", "query_rewrite_consistency" = "LOOSE", "storage_medium" = "HDD", also trace reason mv is blank for the same query
r
The fact that 
TRACE REASON MV
is blank while you are seeing "Failed to init connector" errors is a strong signal that the StarRocks optimizer is failing to even identify your Materialized View (MV) as a candidate for the query. When the trace output is empty, it typically means the optimizer did not find any MVs that overlap with the tables in your query. In your specific case, this is likely caused by the Iceberg catalog failure. 1. Why 
TRACE REASON MV
is blank Usually, if an MV is rejected for being stale or inactive, the trace will say so (e.g., 
MV rewrite fail for mv1: MV is stale
). If it is blank, it means the optimizer didn't even "see" the MV as a match for your query. * Table Resolution Failure: Because your Iceberg connector is failing to initialize (
Failed to init connector
), the FE (Front End) may be unable to correctly resolve the identity of the base tables. If the optimizer cannot confirm that the table in your query is the exact same table used in the MV definition, it will not consider the MV a candidate. * Metadata Missing: Even with 
LOOSE
consistency, the optimizer still needs to load the MV's plan and the base table's schema/partition info. If the Iceberg metadata crawler is stuck (the 
msg: null
error), that metadata may be missing or corrupted in the FE's memory. 2. The Impact of the Iceberg "Reload Failed" Error You mentioned other Iceberg queries work, but the "reload" fails. StarRocks uses a background refresh to poll Iceberg for snapshot changes. * If this background task fails with 
msg: null
(often a 
NullPointerException
), the FE's internal "Catalog Object" enters an unhealthy state. * While direct queries might still work (by using the last known good connection or a direct BE-to-Iceberg path), the MV Rewrite logic relies heavily on the FE's cached metadata of the external catalog. If that cache cannot be refreshed or initialized, the rewrite logic often defaults to "skip" rather than "fail the query." 3. Check for "Inactive" Status Even if you can query the MV directly, it might be marked as inactive for rewrite purposes due to the earlier failures. An inactive MV is ignored by the optimizer. Check status:
Copy code
sql
SELECT name, is_active, last_refresh_error FROM information_schema.materialized_views WHERE name = 'your_mv_name';
If 
is_active
is 
false
, you must reactivate it:
Copy code
sql
ALTER MATERIALIZED VIEW your_mv_name ACTIVE;
4. Troubleshooting Steps 1. Verify Rewrite is Enabled: Ensure the session variable didn't get toggled off:
Copy code
sql
    SHOW VARIABLES LIKE '%enable_materialized_view_rewrite%';
    -- Should be 'on'
2. Fix the Catalog Init Error: * This is the root of the problem. Check the 
fe.log
for the full stack trace of the 
Failed to init connector ... msg: null
error. * If you are using a REST Catalog or AWS Glue, verify your credentials/tokens. If they expired, the background reload will fail, breaking the metadata link required for rewrites. 3. Try `TRACE LOGS MV`: Instead of 
REASON
, use `LOGS`:
Copy code
sql
    TRACE LOGS MV <your_query>;
This provides a much more verbose output of the optimizer's internal steps. If it's still blank, it confirms the optimizer isn't finding a table match. 4. Manual Catalog Refresh: Try to force the catalog to heal:
Copy code
sql
    REFRESH EXTERNAL CATALOG iceberg;
If this returns the same 
msg: null
error, the rewrite will never work until the catalog configuration or the underlying connection is fixed. Summary: The rewrite is failing because the broken Iceberg connector prevents the StarRocks optimizer from "linking" your query tables to your MV tables in its internal metadata map. Fix the 
msg: null
error in the FE logs first. References • data_source/catalog/iceberg/iceberg_catalog.md
Open in Slack
PreviousNext
Powered by