In roadmap for 2025 it lists one of the action items as "runtime filter pushdown to the storage layer". I am using iceberg data lake with 100s of tables with starrocks compute nodes and doing a union all across 100 tables query and joining this with a filtered lookup table that is first filtered with a highly selective predicate on the right side. If right side after filter only matches 10 rows, will starrocks be able to pushdown this predicate condition to each branch of the union all and be able to push the runtime filter to the storage layer of each table so that it can take advantage of iceberg statistics like min/max and scan pushdown to parquet bloom filter etc.? Can you point me to what works currently and what is being worked on, if there is a relevant issue on GitHub I can look into implementing this too myself

@Shaeq Ahmed I'm not quite sure about what exactly he means in this roadmap https://github.com/StarRocks/starrocks/issues/55526. Maybe it's related to some specific cases that runtime filter does not work well with storage later. Generally speaking, we've already supported to push runtime filter to scan node, and scan node can utilize this runtime filter to prune data. runtime filter has been supported since 2022.

@Yan Zhang When testing a join with a constant values cte or a in-subquery the performance is slow and when testing on a iceberg table with one file the iceberg scan node does not prune the scan range using the predicate to narrow down the row/col ranges for page level predicate pushdown and late materialization enabled. For example, in this select star query all rows and columns and the entire file is projected in my test. https://pastebin.com/jsYSvCZ1

can you share the the output of "explain verbose <SQL>". runtime filter will be showed that 1. which node generates runtime filter 2. and which node consumes runtime filter.

19:HASH JOIN
  |  join op: INNER JOIN (PARTITIONED)
  |  equal join conjunct: [18: ps_suppkey, BIGINT, true] = [10: s_suppkey, BIGINT, true]
  |  build runtime filters:
  |  - filter_id = 3, build_expr = (10: s_suppkey), remote = true ///<----- generates
   |  output columns: 1, 3, 11, 12, 14, 15, 16, 20, 23
  |  can local shuffle: true
  |  cardinality: 64800000

8:IcebergScanNode
     TABLE: zz_iceberg_tpch_sf100_iceberg_parquet_lz4.supplier
     TABLE VERSION: Snapshot@(4510802529630325835)
     cardinality=1000000
     avgRowSize=7.0
     dataCacheOptions={populate: true}
     partitions=1/1
     cardinality: 1000000
     probe runtime filters:
     - filter_id = 2, probe_expr = (13: s_nationkey) //<----- consumes