@Rocky looking at grafana, my vacuum errors are static over time, why?

In StarRocks shared-data (lake) clusters, Vacuum Errors in Grafana typically refer to the

starrocks_fe_lake_vacuum_failed_total

metric. This metric tracks the number of times the system failed to delete old, unneeded data files from your object storage (e.g., S3, OSS). If you see these errors as "static" (a flat line at a non-zero value), here is why and what you should check: 1. Cumulative Counters vs. Error Rates Most StarRocks metrics are cumulative counters. A counter only increases when a new error occurs. * Static non-zero value: If the graph is a flat line at a value like

10

, it means 10 errors occurred at some point in the past, but no new errors are currently happening. This is normal Prometheus behavior. * Recommendation: Change your Grafana panel query to use

rate()

or

increase()

, for example:

rate(starrocks_fe_lake_vacuum_failed_total[5m])

If this resulting value is

0

, your cluster is currently healthy and the "errors" you see are just historical baggage. 2. Check the FE Leader Logs If the rate of errors is actually positive and constant (meaning errors are continuously occurring), you need to check the logs of the Leader FE node. Only the Leader FE initiates vacuum tasks. * Search keyword:

lake vacuum

or

Vacuum Deletes

. * Common Causes: * Permissions: The FE lacks

s3:DeleteObject

permissions for the specific bucket/path. * Connectivity: Network issues between the FE and the object storage endpoint. * File Already Deleted: Occasionally, if a file was manually removed or vacuumed by a previous leader, the system may log a failure when it tries to delete it again. 3. Relevant Configurations If you are seeing vacuum-related issues (like storage usage not decreasing), check these FE configurations: * `lake_autovacuum_grace_period_minutes`: Controls how long historical data is kept before it's eligible for vacuuming (default is 30 mins). * `lake_autovacuum_parallel_partitions`: The maximum number of partitions vacuumed simultaneously. Summary: If the value is flat and not increasing, it's a historical count and likely not a current issue. If it is actively increasing, check the Leader FE's

fe.log

for specific S3/object storage error messages. References • faq/shared_data_faq.md • administration/management/monitoring/metrics-shared-data.md • administration/management/monitoring/metrics.md • administration/management/FE_configuration.md