@Rocky On what cases set low_cardinality_optimize_on_lake to true? We using Hive metastore and iceberg over HDFS

The system variable

low_cardinality_optimize_on_lake

(introduced in StarRocks v3.5.0) is used to enable automatic global dictionary optimization for data lake queries (Hive, Iceberg, Hudi, and Delta Lake). In your setup (Hive Metastore and Iceberg over HDFS), you should ensure this is set to

true

(which is its default value) in the following cases: 1. High-Performance String Analytics If your queries frequently perform

GROUP BY

, filtering, or joins on

VARCHAR

or

STRING

columns. This optimization allows StarRocks to process these strings as integers using a dictionary, which is significantly faster and uses less CPU and memory than processing raw strings. 2. Presence of Low-Cardinality Columns It is specifically designed for columns with a low number of unique values (low cardinality), such as: * Status/Flags:

order_status

,

is_deleted

,

gender

. * Categories:

category_id

,

product_type

,

region_code

. * Dates/Time Buckets: If stored as strings (though not ideal), like

YYYY-MM-DD

. 3. Using Parquet or ORC Formats Since you are using Iceberg and Hive, your data is likely in Parquet or ORC format. These formats natively support dictionary encoding. When this variable is

true

, StarRocks' data lake engine can leverage these file-level dictionaries to build a global dictionary automatically during the query, avoiding the need for manual dictionary management. Summary of Benefits for Your Setup: * Reduced I/O and CPU: By reading and processing integer IDs instead of full string values from HDFS. * Faster Aggregations:

GROUP BY

operations on encoded integers are much more efficient than on strings. * Seamless Integration: It works automatically with Hive and Iceberg catalogs without requiring you to manually collect statistics or modify the source data. When to disable it (

false

)? You should generally leave it

true

. The only cases where you might set it to

false

are: * High Cardinality: If the columns have nearly unique values (e.g.,

user_id

or

timestamp

strings), the dictionary overhead might outweigh the benefits. However, StarRocks typically detects this and skips the optimization for such columns. * Memory Pressure: In extremely rare cases, building global dictionaries for a very large number of columns simultaneously might increase FE/BE memory usage. * Troubleshooting: If you suspect a query result mismatch or a performance regression specifically related to string processing after upgrading to v3.5.0. Recommendation: Since you are on v3.5.0+, keep it at its default value of

true

to benefit from the performance gains in your Iceberg/Hive environment. References • table_design/indexes/Bitmap_index.md • sql-reference/System_variable.md • using_starrocks/accelerate-cardinality-preserving-joins.md