http://starrocks.io logo
Join Slack
Powered by
<@U0A71G31CDV> ```with error: starlet err grpc.Ge...
# questions-and-troubleshooting
s
@Rocky
Copy code
with error: starlet err grpc.GetShard(shardId=8890614) error: Deadline Exceeded
2026-06-30 16:33:00.491Z WARN (autovacuum-pool1-t1|4071) [AutovacuumDaemon.vacuumPartitionImpl():280] Vacuumed sqlmesh__contexts.contexts__c_public_user__727550184.8890581 with error: starlet err grpc.GetShard(shardId=8890582) error: Deadline Exceeded
2026-06-30 16:33:01.069Z WARN (autovacuum-pool1-t6|4076) [AutovacuumDaemon.vacuumPartitionImpl():280] Vacuumed sqlmesh__contexts.contexts__c_public_user__727550184.8890701 with error: starlet err grpc.GetShard(shardId=8890702) error: Deadline Exceeded
2026-06-30 16:33:01.128Z WARN (autovacuum-pool1-t8|4078) [AutovacuumDaemon.vacuumPartitionImpl():280] Vacuumed sqlmesh__contexts.contexts__c_public_user__727550184.8890677 with error: starlet err grpc.GetShard(shardId=8890678) error: Deadline Exceeded
2026-06-30 16:33:06.086Z WARN (autovacuum-pool1-t6|4076) [AutovacuumDaemon.vacuumPartitionImpl():280] Vacuumed sqlmesh__contexts.contexts__c_public_user__727550184.8890717 with error: starlet err grpc.GetShard(shardId=8890719) error: Deadline Exceeded
2026-06-30 16:33:06.175Z WARN (autovacuum-pool1-t8|4078) [AutovacuumDaemon.vacuumPartitionImpl():280] Vacuumed sqlmesh__contexts.contexts__c_public_user__727550184.8890733 with error: starlet err grpc.GetShard(shardId=8890734) error: Deadline Exceeded
r
The error 
starlet err grpc.GetShard(shardId=...) error: Deadline Exceeded
occurring in the 
AutovacuumDaemon
logs indicates that the StarRocks Frontend (FE) is timing out while trying to retrieve metadata for data shards (tablets) in a shared-data (cloud-native) cluster. Root Cause This error is typically caused by lock contention in the StarManager (StarMgr) component on the FE. StarMgr manages shard metadata using a global read-write lock. 1. Lock Contention: Background operations (like 
Autovacuum
) or cluster events (like a Compute Node (CN) heartbeat failure or restart) can trigger a batch update of shard metadata. These updates hold a write lock on the shard manager. 2. Blocked Requests: While the write lock is held, all other requests needing a read lock (including 
GetShard
calls from the vacuum daemon or queries) are blocked. 3. Timeout: If the lock is held longer than the configured gRPC timeout (default is 5 seconds), the request fails with 
Deadline Exceeded
. Recommended Troubleshooting & Fixes 1. Increase the StarManager gRPC Timeout If your cluster has a large number of tablets or high metadata churn, the default 5-second timeout may be too short. You can increase this in your FE configuration (
fe.conf
):
Copy code
properties
starmgr_grpc_timeout_seconds = 10  # Increase from 5 to 10 or higher
Note: This requires an FE restart to take effect, or can be set dynamically via 
ADMIN SET FRONTEND CONFIG
if the parameter is mutable in your version. 2. Check for Node Instability Check your FE logs (
fe.log
and 
fe.warn.log
) for messages related to 
HeartbeatManager
or 
ShardManager.updateShardReplicaInfo
. * If CN nodes are frequently flapping (connecting/disconnecting), the constant metadata updates will keep the StarManager locked. * Ensure there is no clock skew between FE and CN nodes (max allowed is typically 5s). 3. Control AutoVacuum Parallelism If the 
AutovacuumDaemon
is overwhelming the system, you can reduce the number of partitions it processes in parallel. In FE configuration:
Copy code
properties
lake_autovacuum_parallel_partitions = 4  # Default is 8
4. Investigate "SQLMesh" Table Metadata The logs mention a specific table/partition: 
sqlmesh*contexts.contexts*c_public_user__727550184
. * Check if this table has an excessive number of buckets. A high bucket count (thousands) creates a massive number of shards, which increases metadata pressure and lock hold times during vacuuming. * You can check the bucket count using: 
SHOW CREATE TABLE sqlmesh*contexts.contexts*c_public_user__727550184;
Summary The 
AutovacuumDaemon
is failing to clean up old data because it can't get the shard details in time. While this is a 
WARN
log and the system will retry, persistent errors can lead to "storage bloat" (old files not being deleted from S3/GCS). Increasing 
starmgr_grpc_timeout_seconds
is the most common immediate fix. References • administration/management/FE_configuration.md • faq/operation_maintenance_faq.md • administration/management/BE_configuration.md
2 Views
Open in Slack
PreviousNext
Powered by