@Rocky Currently, when a partition changes, the MV refreshes the entire affected partition. Is full incremental refresh not supported yet?

As of the current StarRocks versions (including v3.3 and the upcoming v4.x series), "full incremental refresh" (at the row or delta level) is now supported but comes with specific requirements and limitations compared to the traditional partition-level refresh. 1. The Two Refresh Modes StarRocks distinguishes between two types of incremental behavior: * Partition-level Incremental (Default): This is the behavior you are currently experiencing. StarRocks tracks which partitions have changed and refreshes those specific partitions. However, it still re-calculates the entire partition by performing an

INSERT OVERWRITE

on it. * Row-level Incremental (IVM - Incremental View Maintenance): Introduced as a major feature (starting with a preview in v3.3), this mode processes only the deltas (newly appended rows). It appends these deltas to the MV without overwriting or reprocessing existing data in the partition. 2. How to Enable Row-level Incremental Refresh To use the "full" incremental mode, you must explicitly set the

refresh_mode

property when creating the Materialized View:

sql
CREATE MATERIALIZED VIEW my_incremental_mv
REFRESH ASYNC
PROPERTIES (
    "refresh_mode" = "INCREMENTAL"
)
AS 
SELECT ... FROM base_table;

3. Current Limitations While row-level incremental refresh significantly reduces compute costs, it is currently more restricted than the partition-based mode: * Append-only Only: In its current phase, row-level incremental refresh generally only supports append-only data changes. If the base table undergoes

UPDATE

,

DELETE

, or

MERGE

operations, the incremental refresh task may fail or fall back to a full refresh. * Operator Support: It supports common operators like

SELECT

,

WHERE

, and

JOIN

. However, complex aggregations or specific analytical functions may not yet be compatible with this mode. * Metadata Overhead: For external tables (like Iceberg or Paimon), this mode relies on the source format's snapshot/binlog capabilities to identify the exact delta rows. Summary If your goal is to avoid refreshing the entire partition when only a few rows change: 1. Check if your workload is append-only. 2. Try recreating your MV with

"refresh_mode" = "INCREMENTAL"

. 3. If your query contains complex logic that doesn't support row-level IVM yet, you will have to stick with the partition-level refresh. In that case, you can minimize the impact by using finer-grained partitioning (e.g., partitioning by hour instead of by day) to reduce the amount of data reprocessed per refresh. References • sql-reference/sql-statements/materialized_view/CREATE_MATERIALIZED_VIEW.md • using_starrocks/async_mv/use_cases/create_partitioned_materialized_view.md • using_starrocks/async_mv/use_cases/data_lake_query_acceleration_with_materialized_views.md