@Rocky Starrocks 3.5.10 - shared-data mode. When attempting to run REFRESH MATERIALIZED VIEW my_mview WITH SYNC MODE I am getting execute task mv-12112755 failed: Refresh mv my_mview failed after 1 times, try lock failed: 0, error-msg : java.lang.NullPointerException What can I do?

The error message you are seeing—specifically the combination of

try lock failed: 0

and `java.lang.NullPointerException`—indicates that the Materialized View (MV) refresh task crashed during its initial execution phase in the StarRocks task scheduler. In version 3.5.10, several

NullPointerException

(NPE) bugs related to Materialized View refreshes were identified, particularly in shared-data mode or when dealing with complex partition metadata and statistics. Why this is happening 1. "try lock failed: 0": This is a generic wrapper message from the StarRocks Task Scheduler. It means the task failed on its very first attempt (

0

retries/lock attempts) because an unhandled exception (the NPE) occurred before the task could proceed with its normal lifecycle. 2.

java.lang.NullPointerException

: The actual root cause. In 3.5.10, this is frequently caused by: * Missing Statistics: A known bug (#61935) where the refresh logic hits an NPE when trying to fetch partition-level column statistics during the planning stage if that data is missing. * Partition Metadata Inconsistency: In shared-data mode, if there is a discrepancy between the FE's cached metadata and the actual data in object storage (e.g., after a warehouse change or base table modification), the refresh processor may encounter a null reference. * External Catalog Issues: If your MV is built on an external catalog (like Iceberg or Hive), certain partition mapping logic in 3.5.10 can trigger an NPE if partition columns contain NULL values or have specific naming conventions. Recommended Actions 1. Upgrade to StarRocks 3.5.11 or 3.5.12 (Highly Recommended) Several critical NPEs in the MV refresh and planning logic were fixed in the subsequent patches for the 3.5 branch. * Issue #61935 (NPE during stats fetching) was fixed in 3.5.11. * Issue #63043 and #62796 (other NPEs related to metadata/statistics) were also addressed in 3.5.11/3.5.12. Upgrading to the latest 3.5.x patch is the most reliable way to resolve this. 2. Refresh Statistics Manually If you cannot upgrade immediately, try manually collecting statistics on the base table. This often populates the missing metadata that causes the NPE:

sql
ANALYZE TABLE <base_table_name>;

3. Diagnose via

information_schema

Check for more detailed error information in the system views. Even though

SYNC MODE

returned the error, the background task record might have a more descriptive `EXTRA_MESSAGE`:

sql
SELECT * FROM information_schema.task_runs 
WHERE TASK_NAME = 'mv-12112755' 
ORDER BY CREATE_TIME DESC LIMIT 1\G

Look at the

ERROR_MESSAGE

and

EXTRA_MESSAGE

fields. If you see a stack trace referencing

com.starrocks.sql.optimizer.statistics

, it confirms the statistics bug. 4. Verify MV Activity Sometimes an MV becomes

INACTIVE

after a failed refresh or a schema change on the base table. Try manually setting it to active before refreshing:

sql
ALTER MATERIALIZED VIEW my_mview ACTIVE;
REFRESH MATERIALIZED VIEW my_mview WITH SYNC MODE;

5. Check for NULLs in Partition Keys If your MV is partitioned, ensure that the base table's partition column does not contain

NULL

values, as this was a known trigger for refresh failures in shared-data clusters in early 3.5 releases. References • sql-reference/sql-statements/materialized_view/REFRESH_MATERIALIZED_VIEW.md • administration/management/resource_management/Replica.md • deployment/shared_data/azure.md • using_starrocks/async_mv/Materialized_view.md

@Rocky I am hearing that underlying storage rant out of space. Any emergency cleanup commands available?

In a StarRocks shared-data (compute-storage separation) cluster, "running out of space" can refer to two different locations: the local disk cache on your Compute Nodes (CNs) or the remote object storage (S3, OSS, etc.). Depending on which one is full, here are the emergency actions you can take: 1. If Local Disk Cache is Full (CN/BE nodes) The local cache is where StarRocks stores hot data for query acceleration. If these disks are full, it can impact performance or cause IO errors. * Emergency Clear (Non-disruptive): You can temporarily set the cache quota to 0 to force a cleanup, then set it back to your desired limit.

sql
    -- Replace <BE_ID> with your specific BE/CN ID
    UPDATE information_schema.be_configs
    SET VALUE = "0"
    WHERE NAME = "datacache_disk_size" AND BE_ID = <your_be_id>;

    -- After it cleans up, set it back (e.g., to 2TB)
    UPDATE information_schema.be_configs
    SET VALUE = "2T"
    WHERE NAME = "datacache_disk_size" AND BE_ID = <your_be_id>;

* Emergency Clear (Manual): If the nodes are unresponsive, you can delete the

datacache

directory on the physical disk and restart the CN service.

bash
    rm -rf /path/to/storage_root_path/datacache/*

2. If Remote Object Storage is Full (S3/OSS/etc.) This is more critical. Since shared-data clusters use a multi-versioning system, physical deletion of data files often lags behind SQL

DELETE

or

DROP

commands. * Force Drop Tables: If you need to delete a table and free up space immediately, use

FORCE

. This bypasses the recycle bin.

sql
    DROP TABLE table_name FORCE;

* Reduce Trash Retention: Shorten the time StarRocks keeps "deleted" files in the recycle bin.

sql
    -- Set FE recycle bin to 1 hour (default is usually 86400s/24h)
    ADMIN SET FRONTEND CONFIG ("catalog_trash_expire_second" = "3600");

* Trigger Manual Compaction: Compaction merges many small files into fewer large ones and marks the old versions as ready for "Vacuuming" (physical deletion).

sql
    ALTER TABLE table_name COMPACT;

* Check Vacuum Status: In shared-data mode, a background process called "Vacuum" physically deletes unreferenced files from object storage. If it's lagging, you may see high storage usage. You can monitor it via:

sql
    -- Check for backlogged vacuum tasks
    SHOW PROC '/compactions';

3. Cleanup of Orphaned "Garbage" Data Sometimes failed loading tasks or aborted compactions leave "orphaned" files in object storage that the system doesn't know about. StarRocks provides a specific tool for this called

lake_datafile_gc

. * Tool Path: It is usually located in the

bin/

directory of your StarRocks installation as part of the

meta_tool.sh

. * Usage Example:

bash
    ./bin/meta_tool.sh --operation=lake_datafile_gc \
      --root_path="<s3://your-bucket/path>" \
      --expired_sec=3600 --do_delete=true

_Note: Always run with

--do_delete=false

first to see what would be deleted._ Summary Recommendation for the NPE Error Regarding your original NullPointerException, it is very likely triggered because the system is trying to write metadata or refresh a view while the underlying storage is in a "Read-Only" or "Full" state. 1. Clear some space using the commands above. 2. Upgrade to 3.5.11+ if possible, as several NPE bugs during MV refreshes were fixed in that version. 3. Run

ANALYZE TABLE <base_table>

once space is cleared to fix any corrupted/null statistics that might be causing the refresh to fail. References • knowledge/admin_and_operation/shared_data_garbage_cleanup.md • knowledge/admin_and_operation/shared_data_compaction.md

@Rocky - Is there any kind of automated retention schedule for cleanups of old materialized view fragments?

Yes, StarRocks has several automated retention mechanisms specifically for materialized views (MVs) to manage both the metadata (task history) and the data fragments (partitions/files). Since you are on 3.5.10, you have access to the latest TTL (Time-To-Live) features. Here is how you can manage retention: 1. Data Retention (Automated Partition Cleanup) If your MV is partitioned, you can automate the deletion of old data "fragments" (partitions) using TTL properties. StarRocks will check and prune old partitions during each refresh cycle. *

partition_ttl

: Retains data within a specific time window.

sql
    ALTER MATERIALIZED VIEW my_mview SET ("partition_ttl" = "1 MONTH");

*

partition_ttl_number

: Retains a specific number of the most recent partitions.

sql
    ALTER MATERIALIZED VIEW my_mview SET ("partition_ttl_number" = "30");

*

partition_retention_condition

(New in 3.5.x): Allows for more complex cleanup logic.

sql
    ALTER MATERIALIZED VIEW my_mview SET ("partition_retention_condition" = "dt >= CURRENT_DATE() - INTERVAL 3 MONTH");

2. Task History Retention (Metadata Cleanup) Every time an MV refreshes, it creates a record in the task history. If these accumulate, they can consume FE memory and disk space. These are controlled by global FE configurations: *

task_runs_ttl_second

: (Default: 7 days) How long the history of MV refreshes is kept in

information_schema.task_runs

. *

task_runs_max_history_number

: (Default: 10,000) The maximum number of historical records to keep. *

enable_task_history_archive

: If enabled, old tasks are moved to an internal table rather than just being deleted. To adjust these, use:

sql
ADMIN SET FRONTEND CONFIG ("task_runs_ttl_second" = "86400"); -- Keep only 1 day

3. Shared-Data "Fragment" Cleanup (Object Storage) In shared-data mode, even if you delete a partition, the physical files (fragments) might remain in object storage for a "safety" period. * Recycle Bin: Deleted table data is kept in a trash folder for a period defined by

catalog_trash_expire_second

(default is 24 hours). If you are in a storage emergency, you can lower this value to trigger faster physical deletion. * Vacuuming: StarRocks automatically runs a "Vacuum" process to physically delete unreferenced data files. In 3.5.x, this is generally automated, but you can check the status of these background jobs via:

sql
    SHOW PROC '/compactions';

Summary of Emergency Commands If you need to free up space immediately due to your storage issue: 1. Lower the Trash TTL:

ADMIN SET FRONTEND CONFIG ("catalog_trash_expire_second" = "600");

(10 minutes). 2. Force Prune MV Partitions: Set a very short

partition_ttl

on your largest MVs. 3. Drop Unused MVs with FORCE:

DROP MATERIALIZED VIEW <name> FORCE;

(The

FORCE

keyword bypasses the recycle bin). References • using_starrocks/async_mv/use_cases/create_partitioned_materialized_view.md • table_design/data_distribution/expression_partitioning.md • using_starrocks/async_mv/use_cases/data_modeling_with_materialized_views.md

@Rocky Wouldn't

partition_retention_condition and partition_ttl lead to loss of data?

To clarify, using

partition_ttl

or

partition_retention_condition

on a Materialized View (MV) will not cause a permanent loss of your raw business data. Here is the distinction between what is deleted and what is preserved: 1. "Loss of Materialization" vs. "Loss of Data" When a partition in an MV falls outside of your TTL (Time-To-Live) window, StarRocks deletes the pre-computed results for that partition within the MV itself to save storage. However: * Base Tables are Untouched: The raw data in your source tables (the tables you used in the

SELECT

part of the MV) remains exactly as it is. * Queries Still Work: If a user runs a query that needs data from a "deleted" partition, StarRocks' cost-based optimizer is smart enough to see that the MV no longer has that data. It will automatically route the query to the base table instead. * The Result: You still get the correct answer; the query might just run slightly slower because it isn't using the pre-computed MV. 2. When to Use These Settings These settings are designed to create a "Sliding Window" of performance. * Example: If you set

partition_ttl = "3 MONTHS"

, you are essentially saying: "I want the last 3 months of data to be lightning-fast. For anything older, it's okay if the query takes a few extra seconds to scan the raw tables." 3. Comparison of the Settings Property | Behavior | Use Case

partition_ttl

| Keeps partitions where the date is within

X

amount of time from today. | Standard rolling windows (e.g., "Last 90 days").

partition_ttl_number

| Keeps exactly the last

N

partitions regardless of time. | When you have a fixed "budget" for how many partitions you want to materialize.

partition_retention_condition

| (New in 3.5) Keeps partitions that satisfy a specific SQL expression. | Complex logic (e.g., "Keep all data from 2024, plus the last 30 days of 2023"). Summary for your Storage Emergency If you are running out of space in your shared-data cluster: 1. Applying a

partition_ttl

to your largest MVs is a safe and effective emergency move. 2. It will physically delete the MV's data fragments from object storage during the next refresh cycle. 3. Your analysts will still see the full history of data in their reports (as long as the base table has it), but only the recent window will be accelerated by the MV. Warning: The only way to actually lose the "source" data is if you set a TTL/Retention policy on the Base Table itself (if it is a StarRocks internal table). Setting it on the Materialized View only affects the cache/pre-computation. References • using_starrocks/async_mv/feature-support-asynchronous-materialized-views.md • using_starrocks/async_mv/use_cases/create_partitioned_materialized_view.md