@Rocky what is the status with iceberg v3 deletion vectors for starrocks 4.1 ?

In StarRocks 4.1, the status of Iceberg v3 deletion vectors is that they are currently a work in progress and are not yet fully supported for reading or writing. While StarRocks 4.1 is a major milestone for Iceberg v3 compatibility, it focuses on other core v3 features. Here is the breakdown of the Iceberg v3 and deletion-related status in version 4.1: 1. What IS Supported for Iceberg v3 in StarRocks 4.1: * Row Lineage &

_row_id

: StarRocks 4.1 supports reading the

_row_id

column and leverages row lineage from Iceberg v3 tables. * Global Late Materialization: This version introduces late materialization for Iceberg v3, which significantly improves query performance by delaying the reading of data columns until after filters (including those based on

_row_id

) are applied. * Default Column Values: Support for the Iceberg v3 spec's default value feature (metadata-only column additions). * VARIANT Type: Full support for the new

VARIANT

data type introduced in the Iceberg v3 spec for semi-structured data. 2. Status of Deletes and Deletion Vectors (DVs): * Iceberg DELETE Support: StarRocks 4.1 introduces the ability to perform

DELETE FROM <iceberg_table>

directly. However, it currently writes Position Delete files (the v2 mechanism) rather than v3 Deletion Vectors (Puffin files). * Deletion Vectors (DVs): While StarRocks supports Deletion Vectors for Delta Lake (since v3.4.1) and Paimon (since v3.3), the support for the specific binary Deletion Vector format in Iceberg v3 is still on the roadmap. * Engine Compatibility: At this stage, StarRocks 4.1 acts as a "V3-ready" engine for metadata and row tracking, but it still relies on the v2-style position delete mechanism for row-level modifications. Summary for StarRocks 4.1: Feature | Status Iceberg v3 Metadata/Spec | Supported Iceberg v3

VARIANT

Type | Supported Iceberg v3 Row Lineage | Supported Iceberg

DELETE

Statement | Supported (via Position Deletes) Iceberg v3 Deletion Vectors | Not yet supported (Roadmap item) If your workload specifically requires Iceberg v3 Deletion Vectors (the new binary format stored in Puffin files), you should continue to use v2 position deletes for now or check for updates in the upcoming 4.x minor releases where this feature is expected to be finalized. References • data_source/catalog/iceberg/iceberg_catalog.md • data_source/catalog/iceberg/DML.md • data_source/feature-support-data-lake-analytics.md • sql-reference/sql-statements/table_bucket_part_index/DELETE.md • loading/Load_to_Primary_Key_tables.md

@Rocky PRs and github issues regarding work on iceberg v3 and deletion vectors

Regarding StarRocks 4.1, the implementation of Iceberg v3 deletion vectors is still ongoing. While StarRocks 4.1 is a major "Iceberg v3-ready" release, it currently focuses on reading row lineage and metadata rather than the new binary deletion vector format (Puffin files). Here are the specific GitHub PRs and issues tracking this work: 1. Supported: Row Lineage &

_row_id

StarRocks 4.1 has officially implemented the core row tracking features of the Iceberg v3 spec. This allows StarRocks to uniquely identify rows and track their modification history. * PR #62318:

[Feature] support read iceberg v3's _row_id

. This PR implemented the logic to read the reserved

_row_id

and

_last_updated_sequence_number

metadata columns. * Issue #62580:

support read iceberg v3's _row_id field

. This was the primary feature request tracking the need for

_row_id

to enable optimizations like global late materialization. * Functionality: You can now query these columns directly:

sql
    SELECT _row_id, _last_updated_sequence_number, * FROM iceberg_v3_table;

2. Status of Deletion Vectors (DVs) Although StarRocks 4.1 introduces a

DELETE

statement for Iceberg, it does not yet use v3 Deletion Vectors. * Current Write Behavior: When you run a

DELETE

in StarRocks 4.1, it writes v2-style Position Delete files (Parquet) rather than v3 Deletion Vectors (Puffin). * Read Support: Support for reading Puffin-based deletion vectors is currently on the roadmap but is not included in the initial 4.1 release. * Ongoing Work: Development is moving toward a unified "Deletion Vector" reader that will handle DVs across Iceberg, Delta Lake, and Paimon. While Delta Lake DVs are already supported, the Iceberg-specific Puffin integration is part of the follow-up 4.x minor releases. 3. Other Iceberg v3 PRs in 4.1 * VARIANT Type Support: StarRocks 4.1 added full support for the new Iceberg

VARIANT

type for semi-structured data, which is a key part of the v3 spec. * Late Materialization: Leveraging the work in PR #62318, StarRocks 4.1 uses

_row_id

to perform "Global Late Materialization," which skips reading data for rows that have been filtered out or deleted, significantly boosting performance for Iceberg v3 tables. Summary for Tracking: If you want to track progress on GitHub, follow the updates under the [Feature] Iceberg V3 Support labels. While the metadata and row lineage foundations are merged (#62318), the specific "Deletion Vector" (Puffin) read/write support is expected in upcoming patches or minor versions following 4.1.