@Rocky in starrocks 4.1 shared db, is both row and column level security supported with apache ranger

Yes, in StarRocks 4.1, both Row-Level Security (RLS) and Column-Level Security (CLS) are supported through integration with Apache Ranger. This support applies to both the traditional shared-nothing architecture and the shared-data (cloud-native) architecture, which is a primary focus of the 4.1 release. Key Security Features with Apache Ranger: * Row-Level Security (RLS): You can create "row-level filter policies" in Apache Ranger to restrict the rows a user can see based on specific conditions or user attributes. * Column-Level Security (CLS): This is implemented through "masking policies" in Apache Ranger, allowing you to redact or mask sensitive data in specific columns (e.g., showing only the last four digits of a credit card). * Unified Management: The Ranger plugin allows you to manage permissions for both StarRocks internal tables and external tables (via External Catalogs like Hive or Iceberg) from a single interface. Important Considerations: 1. FE Configuration: To enable this, you must set

access_control = ranger

in your

fe.conf

and configure the StarRocks Ranger plugin on your Frontend (FE) nodes. 2. Kerberos Limitation: StarRocks currently does not support connecting to Apache Ranger servers that use Kerberos for authentication. 3. Policy Enforcement: Authorization checks occur at the FE level before query execution. This ensures that regardless of whether your data is stored locally (shared-nothing) or in object storage (shared-data), the security policies are strictly enforced. If you are using the Data Sharing feature (cross-cluster sharing), note that authorization is typically managed by the cluster initiating the query. If that cluster is integrated with Ranger, it will apply the policies defined in the Ranger service associated with that cluster' with that cluster's cluster' that cluster. References • administration/user_privs/authorization/ranger_plugin.md • best_practices/authentication_authorization.md

@Rocky does os 4.1 shared data for external Iceberg support iceberg v3 table with vector deletions for merge on read

Yes, StarRocks 4.1 supports querying Iceberg v3 tables, including the use of Binary Deletion Vectors for efficient Merge-on-Read (MoR) operations. This support extends to both StarRocks' shared-nothing and shared-data (cloud-native) architectures when accessing external Iceberg catalogs. Iceberg v3 & Deletion Vector Support in 4.1 The Iceberg v3 specification (format-version 3) introduces several key features that StarRocks 4.1 implements: * Binary Deletion Vectors: This is the core "vector deletion" feature you mentioned. In Iceberg v3, deletion vectors provide a more compact and performant way to track deleted rows compared to the position/equality delete files used in v2. StarRocks 4.1 can read these vectors, which significantly improves the performance of Merge-on-Read queries by reducing the overhead of filtering deleted rows at query time. * Row Lineage Tracking: StarRocks 4.1 supports querying new v3 metadata columns such as

_row_id

and

_last_updated_sequence_number

. These allow you to track individual rows even after operations like compaction or

rewrite_data_files

. * Variant Data Type: 4.1 includes native support for the Iceberg Variant type, allowing high-performance queries on semi-structured data stored in the Iceberg v3 format. * Default Column Values: Support for reading and creating tables with the Iceberg v3 default value property. Shared Data Architecture Considerations In the Shared Data architecture (where StarRocks itself stores data in object storage), the engine's ability to query External Catalogs (like Iceberg) remains consistent with the shared-nothing version. The 4.1 release specifically optimizes the data lake analytics experience for shared-data clusters, ensuring that features like deletion vector support and metadata caching are available and performant. Summary Table for Iceberg Deletes in StarRocks Iceberg Version | Delete Mechanism | StarRocks Support v1 | Copy-on-Write (CoW) | Full support v2 | Position / Equality Deletes (MoR) | Supported (v3.1+ for Position, v3.3+ for Equality) v3 | Binary Deletion Vectors (MoR) | Supported from v4.1 onwards To use these features, ensure your Iceberg table is created or upgraded to

format-version = '3'

. References • data_source/catalog/iceberg/iceberg_catalog.md • release_notes/release-4.0.md • data_source/feature-support-data-lake-analytics.md • data_source/catalog/iceberg/DML.md References • data_source/catalog/iceberg/iceberg_catalog.md • release_notes/release-4.0.md • release_notes/release-2.1.md • release_notes/release-3.4.md • data_source/feature-support-data-lake-analytics.md

@Rocky quote where in the docs v3 iceberg is supported in 4.1? Does only say v2 is supported File format Compression format Iceberg table version Parquet SNAPPY, LZ4, ZSTD, GZIP, and NO_COMPRESSION • v1 tables: supported. • v2 tables: supported from StarRocks v3.1 onwards in which queries on these v2 tables support position deletes. In v3.1.10, v3.2.5, v3.3 and their later versions, queries on v2 tables also support equality deletes.

In StarRocks 4.1, support for Iceberg v3 (format-version 3) is officially introduced, but it is partial. While the version 4.1 documentation adds several v3 capabilities, it does not yet support Binary Deletion Vectors for Merge-on-Read (MoR). Where the docs mention Iceberg v3 in 4.1: 1. Release Notes (v4.1): The release notes explicitly list v3 support for specific features: > "*Iceberg v3 Support*: Added support for Iceberg v3 default value feature and row lineage. #69525 #69633." > "*VARIANT Type* for Semi-Structured Data in Iceberg: Supports the VARIANT data type in Iceberg Catalog for flexible, schema-on-read storage and querying... #63639 #66539." 2. Iceberg Metadata Tables (v4.1): The documentation for metadata tables confirms the specific version support: > "From v4.1 onwards, for Iceberg v3 tables (format-version = 3), StarRocks supports querying the following Row Lineage metadata columns:

_row_id

,

_last_updated_sequence_number

." 3. Iceberg DDL Operations (v4.1): The documentation for default values notes: > "This feature requires Iceberg format version 3 (

"format-version" = "3"

)." Why you only see v2 in the "Iceberg table version" table: The table you quoted specifically refers to the supported Merge-on-Read (MoR) mechanisms. Currently, StarRocks only supports v2-style MoR (Position Deletes and Equality Deletes). As of StarRocks 4.1, Binary Deletion Vectors (the v3 MoR mechanism) are not yet supported for queries. In fact, the 4.1 release notes include a "fail fast" mechanism for this exact reason:

"Fails fast for unsupported Iceberg V3 features. #70242"

Summary of Iceberg v3 Support in 4.1 Iceberg v3 Feature | Supported in 4.1? Row Lineage (

_row_id

) | Yes VARIANT Data Type | Yes Default Column Values | Yes Binary Deletion Vectors | No (Fails fast) DML (DELETE/TRUNCATE) | Yes (via Partition Overwrite/CoW) Recommendation: If your Iceberg v3 tables rely on Binary Deletion Vectors for updates and deletes, you should continue to use position deletes (v2) or wait for a future StarRocks update that implements the binary vector scanner. References • releasenotes/release-4.1.md • data_source/catalog/iceberg/iceberg_metadata_tables.md • sql-reference/data-types/semi-structured/VARIANT.md References • data_source/catalog/iceberg/DML.md • data_source/feature-support-data-lake-analytics.md • data_source/catalog/deltalake_catalog.md • data_source/catalog/iceberg/iceberg_catalog.md • sql-reference/sql-statements/table_bucket_part_index/DELETE.md • data_source/catalog/iceberg/DDL.md

@Rocky to clarify does starrocks support the reading of v3 vector deletions? we process data using pyspark into iceberg and want starrocks to be able to read the vector deletion files ofr v3

In StarRocks 4.1, while there is newly added support for several Iceberg v3 features, reading Iceberg v3 Binary Deletion Vectors is not yet supported. If you process data using PySpark into Iceberg v3 tables and those tables utilize the new binary deletion vectors (stored in

.puffin

files) for Merge-on-Read (MoR), StarRocks 4.1 will not be able to process those deletions. In fact, StarRocks 4.1 includes a "fail fast" mechanism (via PR #70242) specifically to detect and reject Iceberg v3 tables that use unsupported features like binary deletion vectors to prevent incorrect query results. What is supported for Iceberg v3 in StarRocks 4.1: StarRocks 4.1 introduced "Iceberg v3 support," but it is limited to the following capabilities: * Row Lineage Tracking: You can query the metadata columns

_row_id

and

_last_updated_sequence_number

. * VARIANT Data Type: Native support for the Iceberg v3 Variant type for semi-structured data. * Default Column Values: Support for reading and creating tables that use the v3 schema-level default values. What is NOT supported: * Binary Deletion Vectors (v3 MoR): StarRocks cannot currently read the bitset-based deletion vectors introduced in the v3 specification. * Querying MoR v3 Tables: If a v3 table has active deletion vectors, StarRocks will fail the query rather than ignore the deletes (which would result in "dirty" reads). Summary of Deletion Support Iceberg Version | Delete Mechanism | StarRocks 4.1 Support v1 | Copy-on-Write (CoW) | Full Support v2 | Position Deletes (MoR) | Full Support v2 | Equality Deletes (MoR) | Full Support v3 | Binary Deletion Vectors (MoR) | Not Supported (Fails fast) Recommendation: For your StarRocks-PySpark pipeline, you should continue to use Iceberg v2 for tables requiring Merge-on-Read, or ensure your v3 tables are compacted (using

rewrite_data_files

) into a state without active deletion vectors before querying them in StarRocks. References • releasenotes/release-4.1.md • data_source/catalog/iceberg/iceberg_catalog.md • data_source/catalog/iceberg/iceberg_metadata_tables.md References • data_source/catalog/iceberg/iceberg_catalog.md • data_source/catalog/iceberg/DML.md • release_notes/release-4.0.md • data_source/feature-support-data-lake-analytics.md • release_notes/release-2.1.md