Varun Jain
10/06/2025, 5:17 PM```CREATE MATERIALIZED VIEW mv.audit_logs_last_30_days
PARTITION BY (date_trunc('day', occurredAt), shardId)
DISTRIBUTED BY HASH(customer_id) BUCKETS 32
ORDER BY (occurredAt)
REFRESH ASYNC EVERY (INTERVAL 5 MINUTE)
PROPERTIES (
"replication_num" = "3",
"partition_refresh_number" = "3",
"partition_retention_condition" = "date_trunc('day', occurredAt) >= CURRENT_DATE() - INTERVAL 1 MONTH"
)
AS SELECT * FROM audit_log_rest_iceberg.audit_log_rest_catalog.al_rest_catalog_lt1;```Every 5 min Base Iceberg Table, get ingested with 3.6 Millions records every 5 min, I have 3 BE Nodes with 6 CPU and 24GB Mem each. I can see CPU% never goes beyond 35% and Mem is also under 40%. In the current setup MV lag is 30min+. How can I improve the MV Refresh performance?
Varun Jain
10/06/2025, 5:18 PMAllen Li
10/06/2025, 5:53 PMselect * from information_schema.task_runs
alter materialized view audit_logs_last_30_days("session.enable_profile"="true");Varun Jain
10/06/2025, 7:16 PMVarun Jain
10/06/2025, 9:37 PMVarun Jain
10/06/2025, 9:37 PMVarun Jain
10/07/2025, 4:44 AM