@Rocky in starrocks 4.1.7 when using variant type -> can it search in iceberg s3 as good a performance as struct (other query engines can, does starrock support this) ?

Yes, StarRocks version 4.1.7 supports the

VARIANT

data type specifically for querying semi-structured data in Apache Iceberg tables (stored in Parquet format with variant encoding). Regarding your performance comparison between

VARIANT

and `STRUCT`: 1. Support & Performance (VARIANT vs. STRUCT) * As Good as STRUCT? For raw extraction speed,

VARIANT

is significantly faster than JSON-as-string because it uses a binary-encoded format that allows for "access-by-offset" (reading specific fields without parsing the whole document). However, compared to a native

STRUCT

, there is a key technical difference in the current implementation: * The "Shredding" Factor: In the Parquet/Iceberg specification,

VARIANT

columns can be shredded (where subfields are stored in separate columns, like a

STRUCT

) or unshredded (stored as a single binary blob). * StarRocks Implementation: As of the v4.1 release, StarRocks currently only supports unshredded variant values for both read and write. * I/O Performance: Because it is unshredded, StarRocks must read the entire binary blob from S3 even if you only query a single nested field. In contrast, a

STRUCT

allows for column pruning, where only the specific sub-field's data is fetched from S3. * Computational Performance: Once the data is in memory, the performance is comparable to

STRUCT

access because of the efficient binary offset indexing. 2. When to Use Which? * Use

STRUCT

: If your schema is rigid and you frequently query specific subfields. This offers the best performance because of S3 I/O pruning (reading less data). * Use

VARIANT

: If your data is highly dynamic (schema-on-read) or evolves frequently. It provides a "middle ground" that is much faster than JSON strings while avoiding the maintenance overhead of frequent

ALTER TABLE

operations required by

STRUCT

. 3. Usage in Iceberg S3 StarRocks 4.1.7 automatically recognizes

VARIANT

columns in Iceberg tables. You can extract values using specialized functions:

sql
SELECT 
    get_variant_string(variant_col, '$.user.name') as name,
    get_variant_int(variant_col, '$.user.age') as age
FROM iceberg_catalog.db.table_name;

Summary: While

VARIANT

in StarRocks 4.1.7 is a major performance leap over JSON, it is not quite "as good" as

STRUCT

for I/O-intensive queries on S3 because it does not yet support subfield "shredding" (pruning). It is, however, the recommended type for semi-structured workloads that require flexibility. For the latest updates on shredding support, you can monitor the StarRocks GitHub repository or check the v4.1 Release Notes. References • sql-reference/data-types/semi_structured/VARIANT.md • data_source/External_table.md • sql-reference/sql-functions/variant-functions/variant_typeof.md

@Rocky when will youl Shredding factor ?

As of StarRocks 4.1.7, there is no officially announced date or specific version number for when shredding (subfield-level columnar storage) for the

VARIANT

type will be released. Based on current development plans and the StarRocks 2026 roadmap, here is the current status: 1. Current Status (v4.1.x) * Unshredded Only: StarRocks 4.1.x supports the

VARIANT

binary format. It is much faster than JSON strings because it uses offset-based lookups, but it still reads the entire binary "blob" from S3. * Initial Support: The primary focus of v4.1 was establishing the

VARIANT

type and ensuring compatibility with the Iceberg V3 specification. 2. Roadmap for Shredding * Planned for 2026: Supporting the "Full Iceberg V3 spec"—which includes shredding—is a key priority in the StarRocks 2026 Roadmap (tracked in GitHub issues like #67632). * Technical Progress: The StarRocks engineering team has indicated that "adaptive shredding" is the next stage of evolution for the

VARIANT

type to close the performance gap with native

STRUCT

types. * Dependency: Since shredding is a relatively new part of the Apache Parquet and Iceberg V3 specifications, the StarRocks implementation is being aligned with these upstream community standards to ensure cross-engine compatibility (e.g., ensuring a table shredded by Spark or Trino can be read by StarRocks). 3. What to Expect When shredding is implemented: * I/O Pruning: StarRocks will be able to read only the specific sub-columns requested in a query (e.g., just

variant_col.user.id

) rather than the entire

VARIANT

blob. * Statistics: Shredding will allow StarRocks to store min/max statistics for nested fields, enabling data skipping at the file level. Recommendation: If you require the absolute best performance on S3 right now and your schema is stable, stick with

STRUCT

. If you need the flexibility of

VARIANT

and can tolerate reading the full blob for now, you are well-positioned to benefit from shredding "automatically" once the feature is released in a future StarRocks update. For the most up-to-date tracking, you can follow the StarRocks GitHub Issue #55949 which discusses the Iceberg Variant implementation. References • sql-reference/data-types/semi_structured/VARIANT.md • data_source/feature-support-data-lake-analytics.md • deployment/shared_data/feature-support-shared-data.md • table_design/data_distribution/feature-support-data-distribution.md