@Rocky I am getting this error on an MV partition write

com.starrocks.common.StarRocksException: Dict Decode failed, Dict can't take cover all key :0: BE:552933

Is there away to avoid it with a different compression?

This error,

Dict Decode failed, Dict can't take cover all key

, is a known issue related to StarRocks' Low-Cardinality String Optimization (Global Dictionary). It occurs when the execution engine tries to decode a value using a dictionary that is incomplete or inconsistent with the data being written or read. This typically happens during Materialized View (MV) refreshes or

INSERT INTO

operations, especially if there are concurrent writes or frequent

TRUNCATE

operations on the source tables. Is it related to compression? While dictionary encoding is a form of compression, changing the physical storage compression (e.g., from

LZ4

to

ZSTD

) will not fix this specific error. The error originates from the execution-level optimization, not the storage format on disk. How to avoid it The most effective way to avoid this error is to disable the low-cardinality optimization for the session or globally. 1. Disable via Session Variables (Recommended) You can disable this optimization before running your write or refresh command. This forces StarRocks to process the strings directly without using the problematic global dictionary.

sql
-- Disable the optimization for the current session
SET cbo_enable_low_cardinality_optimize = false;

-- For newer versions (3.x+), you might also need:
SET low_cardinality_optimize_v2 = false;

2. Disable Globally If the issue persists across many jobs, you can disable it at the cluster level (requires admin privileges):

sql
SET GLOBAL cbo_enable_low_cardinality_optimize = false;

3. Update Statistics Sometimes this error occurs because the "Global Dictionary" is built based on stale statistics. Running a full collection of statistics might resolve the inconsistency:

sql
ANALYZE TABLE <your_source_table> WITH SYNC;

Root Cause & Versioning This error is often a bug in how the global dictionary is maintained during high-concurrency or specific data patterns. It has been addressed in several bug-fix releases: * If you are on version 3.1.x, upgrade to 3.1.8+. * If you are on version 3.2.x, upgrade to 3.2.3+. * It was also fixed in older branches like 2.5.x in later patches. Summary: Changing

compression="ZSTD"

in your table properties won't solve this. Instead, use

SET cbo_enable_low_cardinality_optimize = false;

to bypass the failing optimization. References • sql-reference/sql-functions/dict-functions/dict_mapping.md • using_starrocks/distinct_values/Using_bitmap.md • faq/Sql_faq.md • sql-reference/sql-functions/dict-functions/dictionary_get.md

@Rocky do I need to enable the session variables before creating the materialized view?

To ensure the session variable effectively prevents the "Dict Decode" error during a Materialized View (MV) partition write, you need to consider how the refresh is being triggered (manually or automatically). Here is how you should apply the variables: 1. For Manual Refresh (Command Line) If you are manually triggering the refresh using

REFRESH MATERIALIZED VIEW

, the session variables must be set before you execute the refresh command in the same session.

sql
-- Apply to the current session
SET cbo_enable_low_cardinality_optimize = false;

-- Trigger the refresh
REFRESH MATERIALIZED VIEW your_mv_name WITH SYNC MODE;

Note: Using

WITH SYNC MODE

ensures the refresh task inherits your current session's settings and waits for completion. 2. For Automatic/Background Refreshes Background refresh tasks do not inherit session variables from the session that created the MV. To apply this fix to background jobs, you have two options depending on your StarRocks version: Option A: Set Global Variable (All Versions) This is the most reliable way to ensure all background refresh tasks respect the setting.

sql
SET GLOBAL cbo_enable_low_cardinality_optimize = false;

Option B: Use MV Properties (StarRocks v3.4+) If you are on version 3.4 or later, you can bake the session variable directly into the MV definition using the

session.

prefix in the properties. This is the cleanest approach as it only affects that specific MV.

sql
CREATE MATERIALIZED VIEW your_mv_name
REFRESH ASYNC
PROPERTIES (
    "session.cbo_enable_low_cardinality_optimize" = "false"
)
AS 
SELECT ... ;

If the MV already exists, you can try to alter it (though some session properties require recreating the MV in earlier versions):

sql
ALTER MATERIALIZED VIEW your_mv_name SET ("session.cbo_enable_low_cardinality_optimize" = "false");

3. During Initial Creation Yes, you should enable the variable before running the

CREATE MATERIALIZED VIEW

statement. While the creation itself is just a metadata operation, StarRocks often triggers an initial refresh immediately. Setting the variable beforehand ensures that first write operation doesn't fail with the "Dict Decode" error. Summary Recommendation 1. Try Session-Level first: Set the variable and run a manual

REFRESH ... WITH SYNC MODE

. 2. If it fails on background runs: Use

SET GLOBAL cbo_enable_low_cardinality_optimize = false;

. 3. Check your version: If you are below 3.1.8 or 3.2.3, consider an upgrade as this is a known bug that was significantly improved in those patch releases. References • using_starrocks/async_mv/Materialized_view.md • using_starrocks/async_mv/materialized_view_task_run_details.md • using_starrocks/async_mv/feature-support-asynchronous-materialized-views.md • sql-reference/System_variable.md