http://starrocks.io logo
Join Slack
Powered by
Hi All, :iceberg: + :starrocks: Question I believe...
# questions-and-troubleshooting
m
Hi All, iceberg + starrocks Question I believe that StarRocks (version 4.0.1) doesn’t utilize Iceberg column statistics during queries (at all). We have a large table (over 250TB) partitioned by hour (ts). The ts column contains only hours and has statistics over it, which have been verified in Iceberg metadata. However, when we perform a simple query to retrieve the maximum and minimum values of the ts column, I observe that the query profile shows a full scan of the ts column instead of checking the Iceberg metadata for the statistics. Is there anything that may help utilize iceberg statistics or is it just a missing feature?
👀 1
е
m
Is there any indication in query profile where I can see it used the feature from above ?
е
not sure, i’m on older version and haven’t digged into it
had a chance to play with 4.0
Copy code
mysql> show variables like 'enable_rewrite_%';
+-------------------------------------------+-------+
| Variable_name                             | Value |
+-------------------------------------------+-------+
| enable_rewrite_bitmap_union_to_bitamp_agg | true  |
| enable_rewrite_groupingsets_to_union_all  | false |
| enable_rewrite_or_to_union_all_join       | false |
| enable_rewrite_partition_column_minmax    | true  |
| enable_rewrite_simple_agg_to_hdfs_scan    | false |
| enable_rewrite_simple_agg_to_meta_scan    | false |
| enable_rewrite_sum_by_associative_rule    | true  |
| enable_rewrite_unnest_bitmap_to_array     | true  |
+-------------------------------------------+-------+
8 rows in set (0.00 sec)

mysql> set enable_rewrite_simple_agg_to_hdfs_scan = true;
Query OK, 0 rows affected (0.01 sec)

mysql> set enable_rewrite_simple_agg_to_meta_scan = true;
Query OK, 0 rows affected (0.00 sec)

explain select count(*) from iceberg.xxx.transactions;
+----------------------------------------------------+
| Explain String                                     |
+----------------------------------------------------+
| PLAN FRAGMENT 0                                    |
|  OUTPUT EXPRS:52: count                            |
|   PARTITION: UNPARTITIONED                         |
|                                                    |
|   RESULT SINK                                      |
|                                                    |
|   4:Project                                        |
|   |  <slot 52> : ifnull(55: sum_count, 0)          |
|   |                                                |
|   3:AGGREGATE (merge finalize)                     |
|   |  output: sum(55: sum_count)                    |
|   |  group by:                                     |
|   |                                                |
|   2:EXCHANGE                                       |
|                                                    |
| PLAN FRAGMENT 1                                    |
|  OUTPUT EXPRS:                                     |
|   PARTITION: RANDOM                                |
|                                                    |
|   STREAM DATA SINK                                 |
|     EXCHANGE ID: 02                                |
|     UNPARTITIONED                                  |
|                                                    |
|   1:AGGREGATE (update serialize)                   |
|   |  output: sum(56: ___count___)                  |
|   |  group by:                                     |
|   |                                                |
|   0:IcebergScanNode                                |
|      TABLE: xxx.transactions       |
|      TABLE VERSION: Snapshot@(7808551201887433419) |
|      cardinality=1                                 |
|      avgRowSize=1.0                                |
+----------------------------------------------------+
output: sum(56: ___count___) is what tells it uses meta scan
for count star the table must not have any delete files, no eq deletes no position deletes
played a little more with enable_rewrite_simple_agg_to_hdfs_scan was puzzled by incorrect OutputRows in explain analyze with optimization on while query result is correct turns out hdfs_scanner decides if it can use meta per data file, so OutputRows gets populated with data rows count that were needed to be scanned due to pos deletes
👀 1
m
Thanks for digging 🙏 Tried with vars you set to true, still 1TB scan just for max/min on partition column + with Iceberg stats
Copy code
SELECT
    /*+ SET_VAR(
    enable_rewrite_simple_agg_to_hdfs_scan=true,
    enable_rewrite_simple_agg_to_meta_scan = true)
      */
    min(ts), max(ts)
FROM our_schema.table;
Attaching query profile
~4m on our, ~640 cores cluster 🙂 I see in query profile that column has full stats? or it's full read ? not clear 🙂
max min over iceberg query profile
@Евгений Шишкин who from the StarRocks masters we can tag do dig further ? Also will open an issue on this
е
not sure, run git blame and see who implemented it i have not had a time to look into it yet
🙏 1
made a pull request yesterday, please check when you have some time
🦾 1
Open in Slack
PreviousNext
Powered by