Hi Team, I have MV that refresh the data from the ...
# questions-and-troubleshooting
v
Hi Team, I have MV that refresh the data from the external iceberg table
```CREATE MATERIALIZED VIEW mv.audit_logs_last_30_days
PARTITION BY (date_trunc('day', occurredAt), shardId)
DISTRIBUTED BY HASH(customer_id) BUCKETS 32
ORDER BY (occurredAt)
REFRESH ASYNC EVERY (INTERVAL 5 MINUTE)
PROPERTIES (
"replication_num" = "3",
"partition_refresh_number" = "3",
"partition_retention_condition" = "date_trunc('day', occurredAt) >= CURRENT_DATE() - INTERVAL 1 MONTH"
)
AS SELECT * FROM audit_log_rest_iceberg.audit_log_rest_catalog.al_rest_catalog_lt1;```
Every 5 min Base Iceberg Table, get ingested with 3.6 Millions records every 5 min, I have 3 BE Nodes with 6 CPU and 24GB Mem each. I can see CPU% never goes beyond 35% and Mem is also under 40%. In the current setup MV lag is 30min+. How can I improve the MV Refresh performance?
@Allen Li Can you please help here?
a
You can have a look of the refresh profile by https://docs.starrocks.io/docs/sql-reference/sql-functions/utility-functions/get_query_profile/
Copy code
select * from information_schema.task_runs 
alter materialized view audit_logs_last_30_days("session.enable_profile"="true");
v
Let me try that.
Here is my Query Profile
Untitled
I can see Bottleneck here is Writing to the StarRocks MV table - See RpcClientSideTime, RpcServerSideTime and WaitWriteTime Reading from iceberg is very quick. External Iceberg Table is partitioned by date and shardID. We are using the same partition for MV as well. 10sec+ are spend in WaitWriteTime -> waiting for write.
đź‘€ 1