@Rocky Is it possible to create indexes for data which is in Iceberg tables? Can StarRocks leverage those indexes while querying? Something similar to Apache Pinot

Yes, StarRocks can provide high-performance queries on Iceberg tables, but the mechanism is different from Apache Pinot. While StarRocks does not support running the

CREATE INDEX

command directly on external Iceberg tables, it leverages several built-in features to achieve similar indexing and acceleration benefits. 1. How StarRocks Accelerates Iceberg (Without Manual Indexes) Instead of manually creating indexes on the remote storage, StarRocks uses these automated strategies: * Data Skipping (Metadata-level): StarRocks leverages Iceberg’s manifest files to perform partition pruning and file-level min/max filtering during query planning. This avoids scanning irrelevant files. * Data Caching (Local Disk): You can enable the Data Cache feature. StarRocks will cache frequently accessed data from Iceberg (S3/HDFS) onto the local NVMe/SSD disks of the Backend (BE) nodes. This provides "index-like" performance by avoiding remote I/O and leveraging StarRocks' internal vectorized format. * CBO Statistics: You can run

ANALYZE TABLE

on Iceberg tables. StarRocks collects column-level statistics (NDV, Min/Max, Null count) and stores them in its internal

_statistics_

database. The Cost-Based Optimizer (CBO) uses this to generate the most efficient execution plans (e.g., choosing the best Join order). 2. The "Pinot-like" Alternative: Materialized Views If you need specific indexes (like Bitmap, Bloom Filter, or Inverted Indexes) for data in Iceberg, the StarRocks best practice is to use Asynchronous Materialized Views (MVs): * You can create an MV based on your Iceberg table. * The data for this MV is stored in StarRocks’ native format, allowing you to use

CREATE INDEX

(Bitmap, Bloom Filter, or Inverted) on the MV itself. * StarRocks can automatically rewrite queries directed at the Iceberg table to hit the indexed Materialized View instead, providing a transparent acceleration layer. 3. Comparison with Apache Pinot Feature | StarRocks (on Iceberg) | Apache Pinot Index Management | Leverages Iceberg native metadata + CBO stats. | Manually managed via table/segment configs. External Indexing | Uses Materialized Views to create a searchable, indexed copy of hot data. | Builds specialized index segments (e.g., Star-tree) on top of deep storage. Data Skipping | Automatic via Iceberg manifests and Parquet row-group stats. | Automatic via internal segment metadata. Ease of Use | Pure SQL-based (

CREATE CATALOG

+

ANALYZE

). | Requires JSON configs and segment management. Summary If your goal is to leverage existing indexes in Iceberg, StarRocks automatically uses the format's metadata (min/max, row-group stats) for data skipping. If your goal is to create new indexes to speed up point lookups or text searches, you should use StarRocks Materialized Views on top of your Iceberg tables, as they allow you to apply StarRocks' full suite of indexing capabilities (Bitmap, Inverted, Vector, etc.) to your data lake. References • table_design/StarRocks_table_design.md • table_design/table_design.md • table_design/indexes/vector_index.md • integrations/data_lakes.mdx • table_design/indexes/indexes.md