@Rocky I noticed that initial creation of a partitioned materialized view consists of sequential tasks despite there being many partitions. Is there a way to scale the build with concurrent tasks?

Hi, I'm Rocky! It's common to see partitioned materialized views (MVs) being built sequentially, especially in newer versions of StarRocks. This is due to a change in the default behavior designed to prevent resource exhaustion, but you can definitely tune it to scale the build. Here is why this happens and how you can scale the build with more concurrency: 1. Why it is Sequential (Default Behavior) Starting from v3.3, the default value for the MV property

partition_refresh_number

was changed from

-1

(all partitions at once) to

1

. This means: * StarRocks refreshes partitions one by one, creating a separate sub-task for each partition. * In

information_schema.task_runs

, you will see many individual tasks running one after another. * Limitation: By design, the StarRocks scheduler only allows one refresh task run per materialized view at a time to prevent race conditions and OOM errors. 2. How to Scale the Build (Concurrency & Parallelism) To speed up the initial build and use more system resources, you can use the following strategies: Increase Batch Size (

partition_refresh_number

) Instead of running one task per partition, you can group many partitions into a single task. While this still shows as "one task" in the scheduler, the SQL query behind it will use the full MPP (Massively Parallel Processing) power of your cluster to process those partitions in parallel.

sql
ALTER MATERIALIZED VIEW your_mv_name 
SET ("partition_refresh_number" = "10"); -- Refreshes 10 partitions in one batch
-- Or set to -1 to refresh all partitions in a single large task
SET ("partition_refresh_number" = "-1");

Increase Query-Level Parallelism You can use the

session.

prefix in your MV properties to pass session variables that control the internal parallelism of the refresh query itself. This allows the single active refresh task to utilize more CPU cores:

sql
ALTER MATERIALIZED VIEW your_mv_name 
SET ("session.pipeline_dop" = "16", -- Increase degree of parallelism (v3.0+)
     "session.query_mem_limit" = "17179869184"); -- Increase memory limit (e.g., 16GB)

Use Adaptive Refresh Strategy Introduced in v3.3, you can set the strategy to

adaptive

. StarRocks will then automatically determine how many partitions to refresh in one batch based on the data volume, optimizing the balance between speed and resource safety.

sql
ALTER MATERIALIZED VIEW your_mv_name 
SET ("partition_refresh_strategy" = "adaptive");

Dedicated Resource Group If your MV is hitting a bottleneck, it might be restricted by the default resource group (

default_mv_wg

). You can assign it to a custom resource group with higher CPU and memory limits:

sql
-- First create a resource group, then:
ALTER MATERIALIZED VIEW your_mv_name 
SET ("resource_group" = "high_perf_rg");

Summary Recommendation For a faster initial build: 1. Temporarily set

"partition_refresh_number" = "-1"

(or a high number like

50

). 2. Set

"session.pipeline_dop"

to a higher value to ensure the query uses all available cores. 3. Once the initial build is done, you can set

partition_refresh_number

back to a lower value (like

1

or

adaptive

) for stable, incremental background refreshes. References • administration/management/FE_configuration.md • using_starrocks/async_mv/materialized_view_task_run_details.md • using_starrocks/async_mv/use_cases/create_partitioned_materialized_view.md • using_starrocks/async_mv/troubleshooting_asynchronous_materialized_views.md