Does StarRocks compute stats for iceberg tables? Can it do parallel computing of stats, sketches etc.. for iceberg tables?

Yes, StarRocks can compute statistics for Iceberg tables, and this includes NDVs and various forms of sketches (such as histograms). Starting from version 3.2, StarRocks introduced the ability to use the

ANALYZE

command on Iceberg tables (within an Iceberg Catalog) to collect statistics for its Cost-Based Optimizer (CBO). Here is a breakdown of what is supported: * NDVs (Number of Distinct Values): StarRocks computes NDVs for columns in Iceberg tables. Internally, StarRocks uses HyperLogLog (HLL) sketches to estimate these values. * Histograms: Since version 3.3, StarRocks supports collecting histograms for external tables, including Iceberg. Histograms are a type of statistical sketch used to handle data skew and provide more accurate selectivity estimation for the optimizer. * STRUCT Subfield Statistics: Version 3.3 also added support for collecting statistics on subfields of

STRUCT

types in Iceberg tables. * Multi-column Joint NDV: Starting from version 3.5, StarRocks supports collecting joint NDV statistics for multiple columns, which helps the optimizer understand correlations between columns. * Basic Statistics: These include

row_count

,

data_size

,

null_count

,

min

, and

max

. How it works: 1. Manual Collection: You can trigger the statistics collection manually using the

ANALYZE TABLE

syntax:

sql
    ANALYZE TABLE <catalog_name>.<db_name>.<table_name> [COLUMNS (col1, col2...)];

2. Storage: The collected statistics (including the HLL-based NDVs and histograms) are stored in StarRocks' internal

_statistics_

database (e.g.,

_statistics_.column_statistics

and

_statistics_.histogram_statistics

), not back into the Iceberg table's own metadata files (like Puffin files). 3. Automatic Collection: StarRocks can also be configured to collect these statistics periodically or automatically based on query patterns. In summary: Yes, StarRocks computes these statistics using its own internal sketch-based engine to ensure high-performance query planning for Iceberg data. References • release_notes/release-3.5.md • release_notes/release-3.3.md