Hi everyone, Large scans on equality-deletes: how to get better pruning? We have Apache Flink jobs writing to AWS Glue catalog in two sinks for the same dataset: • Iceberg table • Delta table We query both from StarRocks. A simple

SELECT * FROM <table> LIMIT 100;

behaves very differently: Iceberg plan

...
6:HASH JOIN
| join op: LEFT ANTI JOIN (BROADCAST)
| equal join conjunct: block_height = block_height
| equal join conjunct: order_in_block = order_in_block
| other join predicates: $data_sequence_number < $data_sequence_number
| limit: 100
|
|----5:EXCHANGE
|
3:IcebergScanNode
   TABLE: datacloud_db.transfers_eth_processed_v3_with_delete_file
   cardinality=1000000000

4:IcebergEqualityDeleteScanNode
   TABLE: datacloud_db.transfers_eth_processed_v3_eq_delete_block_height_order_in_block
   Iceberg identifier columns: [block_height, order_in_block]
...

As you can see Starrocks is trying to resolve the deletes before running the query. This causes a full dataset scan (data + equality delete files) before honoring the

LIMIT

. Delta plan

0:DeltaLakeScanNode
   TABLE: transfers_eth_processed_delta
   cardinality=100
   limit: 100

Here StarRocks seems able to push down the limit and prune aggressively. What can we do on the Iceberg side so that StarRocks can avoid scanning the full dataset for simple selects and small limits? (any recommendations would be welcomed on the write/read/table side)

@Amir Ganiev, that's the way we handle iceberg table with eq-delete files. you can view this hash join node as the iceberg scan node with eq-delete files. we use left anti join to handle eq-delete predicates. so that's why we can not push limit clause down any more. the way we handle delta lake is different: we read and parse deletion-vector files inside this delta lake scan node.

Hi Team! @Yan Zhang Following up on this thread, is there a plan for StarRocks to introduce support for Apache Iceberg V3 including the deletion vectors introduced to Apache Iceberg V3 instead of Equality deletes files in Apache Iceberg V2*?*

@Amir Ganiev AFAIK, it's not on roadmap yet because v3(deletion vector)is not widely adopted yet. Current we put more efforts on v3 variant. But i think to support deletion vector should be not very hard since it's been supported in delta lake.