http://starrocks.io logo
Join Slack
Powered by
<@U0A71G31CDV> - starrocks 3.5.20 in shared-data m...
# questions-and-troubleshooting
a
@Rocky - starrocks 3.5.20 in shared-data mode. A few asynchronous materialized views reading from external iceberg catalog became inactive and then active again. All materialized views are active right now. Is there a way to check what was the error that made them inactive in the first place? If it helps - all the logs have been preserved
r
To troubleshoot why your asynchronous materialized views (MVs) became inactive and were subsequently reactivated in StarRocks 3.5.20, you can use a combination of system tables and FE (Frontend) logs. 1. Check Historical Refresh Errors The most direct way to see past errors is through the 
information_schema.task_runs
table. This table preserves the history of refresh attempts, even if the MV is now active and successful. Run the following query to find failed tasks and their error messages:
Copy code
sql
SELECT 
    TASK_NAME, 
    CREATE_TIME, 
    FINISH_TIME, 
    STATE, 
    ERROR_MESSAGE 
FROM information_schema.task_runs 
WHERE STATE = 'FAILED' 
  AND TASK_NAME LIKE 'mv-%'  -- MV tasks are usually prefixed with 'mv-' followed by the MV ID
ORDER BY CREATE_TIME DESC;
_Note: You can find the 
TASK_NAME
for a specific MV by running 
SHOW MATERIALIZED VIEWS LIKE 'your_mv_name'
. Check the 
last_refresh_error_message
column there as well, though it often only shows the most recent failure._ 2. Search FE Logs for State Changes StarRocks logs state transitions (Active/Inactive) in the 
fe.log
 and 
fe.warn.log
. Since you are on version 3.5.20, your logs likely include the MV name as a prefix for easier debugging. Search your FE logs for the following keywords: * "set inactive": To find the moment and reason the MV was deactivated. * "inactive reason": To see the specific cause (e.g., schema change detected in the Iceberg table). * "active checker": StarRocks has a background thread (
active_checker
) that automatically attempts to reactivate MVs. Logs from this thread will explain why it was able to make the MV active again. Common grep patterns:
Copy code
bash
grep -i "set inactive" fe.log
grep -i "inactive_reason" fe.log
grep -i "active_checker" fe.log
3. Why did they become Active again? In StarRocks 3.1.6+, the FE configuration 
enable_mv_automatic_active_check
is enabled by default. * The Inactive Trigger: Asynchronous MVs (especially those reading from external catalogs like Iceberg) are automatically set to 
inactive
if StarRocks detects a schema change, a table swap, or a base table drop/recreate. * The Re-activation: The background checker periodically (every 60 seconds by default) checks inactive MVs. If the underlying schema is found to be compatible again (or if the transient metadata issue is resolved), it automatically runs an 
ALTER MATERIALIZED VIEW ... ACTIVE
command internally. 4. Common Causes for Iceberg MVs When working with an Iceberg Catalog, MVs often become inactive due to: 1. Schema Evolution: If columns were added, dropped, or renamed in the Iceberg table. 2. Metadata Inconsistency: Transient issues where the StarRocks FE cannot reach the Iceberg metastore (Hive/Glue/REST) or the underlying S3/HDFS storage during a refresh or metadata sync. 3. Partition Changes: Changes to the partition spec in Iceberg that conflict with the MV's partition definition. Recommendation: If this happens frequently, check if there are automated ETL processes in your data lake that are performing 
OVERWRITE
or schema modifications on the source Iceberg tables. You can also check the 
INACTIVE_REASON
column in 
information_schema.materialized_views
immediately after a failure occurs. References • administration/management/FE_configuration.md • using_starrocks/async_mv/troubleshooting_asynchronous_materialized_views.md
In StarRocks Primary Key (PK) tables, the state COMMITTED (but not yet VISIBLE) means the Frontend (FE) has recorded the transaction, but the Backend (BE) is still in the "Apply" phase. This phase is where the BE updates the PK index and generates the delete vector to ensure uniqueness and correct versioning. After a BE restart, especially an ungraceful one, the BE must rebuild or synchronize the PK index for all active tablets before those transactions can be made visible. 1. Is this rebuild speed expected? What limits it? The speed of 150–190 MB/h for a 5-6 GB index is considered very slow. At 100% CPU usage, the bottleneck is likely the computation involved in index merging and hashing, exacerbated by memory-to-disk thrashing. * CPU (The primary bottleneck): Rebuilding the PK index involves heavy CPU tasks: calculating hashes for keys and sorting. Since your CPU is at 100% and 
update_apply
threads are fully busy, the system is maxed out on the compute side. * Memory Limits (
l0_max_mem_usage
): This is the most common reason for slow PK index performance. The 
l0_max_mem_usage
(default ~100MB) limits how much of a single tablet's persistent index can stay in memory. If your index is 6GB and the limit is low, the BE is constantly reading/writing index pages to disk (I/O) and re-calculating hashes, which consumes massive CPU. * Apply Worker Count: Your 8 threads are fully saturated. If the BE has more than 8 physical CPU cores, the thread count is artificially limiting your throughput. 2. Can it be sped up safely without another restart? Yes, most relevant parameters are mutable and can be adjusted via the BE HTTP API without a restart. Step A: Increase the Apply Thread Pool If your BE has more cores (e.g., a 16 or 32-core machine), increase the worker count:
Copy code
bash
curl -X POST http://<be_ip>:<be_http_port>/api/update_config?transaction_apply_worker_count=16
Step B: Increase Memory Allotment for Persistent Index This is the most likely "silver bullet." Increasing this reduces the need for the BE to swap index data to disk during the apply phase.
Copy code
bash
# Increase the per-tablet index memory limit (example: set to 512MB or 1GB)
curl -X POST http://<be_ip>:<be_http_port>/api/update_config?l0_max_mem_usage=1073741824

# Ensure the total PK engine memory limit is sufficient (default is 60%)
curl -X POST http://<be_ip>:<be_http_port>/api/update_config?update_memory_limit_percent=70
Step C: Increase Index Loading Parallelism Introduced in v3.2.0, this controls the threads used to fetch the persistent index:
Copy code
bash
curl -X POST http://<be_ip>:<be_http_port>/api/update_config?get_pindex_worker_count=16
3. Persistent Index: 
false
vs. 
rebuild from scratch
 * 
enable_persistent_index=false
: This forces the entire PK index into memory. * Speed: It would be significantly faster if you have enough RAM. * Risk: If 
(PK_size + overhead) * number_of_tablets
exceeds your BE's available RAM, the BE will OOM (Out of Memory) and crash. Given your index is already 6GB per tablet, disabling persistence is dangerous unless you have very high-memory nodes (512GB+). * Rebuilding from scratch: There is no native "rebuild" command that is faster than the current "Apply" process. Dropping and re-creating the table would technically be a "rebuild," but you would have to reload all data, which is rarely faster than letting the BE finish applying the committed versions. Summary of Recommendations 1. Check Hardware: Confirm if the BE has more than 8 cores. If yes, increase 
transaction_apply_worker_count
. 2. Adjust Memory: Immediately increase 
l0_max_mem_usage
to at least 512MB or 1GB. This will reduce the CPU-intensive I/O operations for the persistent index. 3. Monitor: Watch the 
update_apply_queue_count
and 
update_mem_bytes
metrics. If memory usage stays low while CPU is 100%, the memory limits are definitely the bottleneck.
Warning: Do not restart the BE again while it is in this state, as it will simply restart the rebuild process from the last checkpoint, potentially losing the progress made over the last 24 hours. References • best_practices/primarykey_table.md • administration/management/BE_configuration.md • table_design/table_types/primary_key_table.md
Open in Slack
PreviousNext
Powered by