@Rocky looking at the shared data - starlet grafana dashboard, what are the most key metrics for cluster performance/stability?

In StarRocks shared-data clusters, the Starlet and Shared-data dashboards focus on the interaction between the compute nodes and object storage (S3/GCS/Azure). The following are the most critical metrics for assessing performance and stability: 1. I/O Performance & Latency Since data is stored remotely, I/O efficiency is the biggest performance bottleneck. * fslib read io_latency (Quantile/Average): Measures how long it takes to read data from object storage. Spikes here directly correlate with slow queries. * fslib read throughput / iops: High throughput indicates heavy data movement. If this hits the limits of your object storage provider, you will see performance degradation. * fslib s3 single/multi upload latency: Essential for data loading performance. If upload latency increases, your

INSERT

or

ROUTINE LOAD

tasks will slow down. 2. Cache Efficiency To mitigate remote storage latency, StarRocks uses a local Data Cache (StarCache). * fslib cache hit ratio: This is arguably the most important performance metric. A high hit ratio (e.g., >90%) means most data is read from local disk, bypassing slow network I/O. * Metacache Usage / Misses: Tracks the effectiveness of metadata caching. High miss rates for "Metadata Cache" or "Txn Log Cache" can lead to increased latency in query planning and execution. 3. Cluster Stability & Task Health These metrics indicate if the system is overloaded or failing to maintain data consistency. * Publish Version Queued Tasks: A growing queue here is a major red flag. It indicates the cluster is struggling to commit new data versions, which will eventually cause loading to hang. * Vacuum Errors: Shared-data clusters rely on "Vacuum" tasks to clean up old data in object storage. Errors here lead to storage bloat and potential cost increases. * fslib async tasks (Queue Size): Monitors the queue for asynchronous file operations. If this queue stays high, the Starlet service (which manages storage abstraction) may be saturated. 4. Metadata Operations * Get/Put Tablet Metadata Latency: Measures the speed of interacting with the metadata service. Since shared-data clusters decouple metadata, high latency here slows down every aspect of the cluster, from query planning to schema changes. Summary for Troubleshooting: * Slow Queries? Check

fslib cache hit ratio

and

fslib read io_latency

. * Stuck Loads? Check

Publish Version Queued Tasks

. * High Storage Costs? Check

Vacuum Errors

and

Vacuum Deletes

latency. References • administration/management/monitoring/metrics-shared-data.md • administration/management/BE_configuration.md • using_starrocks/caching/block_cache.md • faq/shared_data_faq.md • quick_start/shared-data.md

@Rocky related - looking at the

errors

section in the shared data general dashboard, how can i see what those errors are?

The "Errors" section in the Shared Data General dashboard typically tracks failures related to background management tasks, most notably Vacuum operations (garbage collection of old data versions in object storage). To see what these errors actually are, you should use the following methods: 1. Check BE Logs (Primary Source) Since the Shared Data architecture relies on the Starlet service running within the Backend (BE) or Compute Node (CN), the specific error messages are recorded in the BE log files. * Log Locations: Check

be/log/be.INFO

or

be.WARNING

on the nodes reporting errors. * Keywords to Search: * `Vacuum`: To find why garbage collection tasks are failing. * `starlet`: To find errors related to object storage abstraction (S3/GCS/Azure). * `lake`: To find errors related to the shared-data storage engine. * `fslib`: To find low-level I/O or connection errors to your object store. 2. Inspect Vacuum Task Status If the errors are in the Vacuum section, it means StarRocks is failing to delete expired files from your object storage. * Check FE Logs: The Frontends (FE) coordinate these tasks. Search

fe.log

for

Vacuum

or

LakeService

. * Backlog check: If the "Vacuum Errors" count is high, also check the Vacuum Deletes QPS and Vacuum Pending Tasks in the same dashboard. A high backlog often leads to timeout-related errors in the logs. 3. Check Metadata Health Shared-data clusters store tablet metadata separately. If the errors are related to "Get/Put Tablet Metadata": * Check the connectivity between your BE/CN nodes and the metadata service (often the FE leader). * Search for

fail to get tablet meta

or

RPC error

in the

<http://be.INFO|be.INFO>

logs. 4. Load & Query Errors If the dashboard shows "Loading Errors" or general "Query Errors": * Loading: Use

SHOW LOAD WHERE LABEL = 'your_label';

and look for the

ErrorLogUrl

. This URL points to a specific file on a BE node (typically under

storage/error_log

) containing the row-level reason for failure. * Queries: For query-specific failures, check

information_schema.audit_log

(if enabled) or search the FE

fe.warn.log

for the specific

QueryId

. Summary Recommendation: Start by running

grep -i "error" be/log/be.INFO

(or

be.WARNING

) on a node where the dashboard indicates failures. This is the fastest way to get the stack trace or error code (e.g., S3 403 Forbidden, 404 Not Found, or Connection Timeout). References • administration/management/monitoring/metrics-shared-data.md • administration/management/logs.md • faq/shared_data_faq.md • administration/management/BE_configuration.md • sql-reference/information_schema/be_compactions.md References • administration/management/logs.md • deployment/deploy_manually.md • administration/management/BE_configuration.md • administration/management/audit_loader.md • faq/loading/Loading_faq.md

@Rocky is stuff like this ok

W20260604 14:47:20.935609 139884595115584 vacuum.cpp:165] Fail to delete: Internal error: starlet err grpc.GetShard(shardId=2616292) error: Deadl
ine Exceeded
E20260604 14:47:20.935860 139659143280192 vacuum.cpp:625] Internal error: starlet err grpc.GetShard(shardId=2616292) error: Deadline Exceeded

These log messages indicate that a background Vacuum task (responsible for cleaning up old data in object storage) failed to communicate with the metadata service (the FE) within the expected time. Specifically, the Backend (BE) tried to call

GetShard

to fetch metadata for a tablet, but the gRPC request timed out after the default 5-second limit. Is this "OK"? * Short term: Yes. Vacuuming is an asynchronous background process. If a single task fails due to a timeout, it will be retried in the next cycle. It does not affect data correctness or currently running queries. * Long term: It needs attention. If these errors persist or the "Vacuum Errors" count in your dashboard keeps climbing: 1. Storage Bloat: Old data versions won't be deleted from your object storage (S3/GCS), leading to higher storage costs. 2. Metadata Pressure: It suggests your FE (Frontends) are under heavy load or experiencing lock contention, which can eventually slow down query planning. Common Causes 1. FE Lock Contention: This is a known issue in some StarRocks versions where maintenance tasks (like a node restarting or scaling) hold a write lock on the

ShardManager

, blocking other

GetShard

requests for several seconds. 2. FE Resource Pressure: If the FE is bottlenecked on CPU or experiencing long GC pauses, it may fail to respond to gRPC requests within 5 seconds. 3. High Metadata Volume: If you have a very large number of tablets/shards and are performing heavy loading/compaction, the metadata service can become slow. What to Check 1. FE Logs: Look at

fe.log

and

fe.warn.log

for keywords like

ShardManager

,

slow lock

, or

Starlet

. Look for messages indicating long lock hold times. 2. Dashboard: Check the FE JVM GC metrics and FE CPU Usage. 3. Configuration: If your FE is healthy but simply slow due to the scale of your cluster, you can try increasing the gRPC timeout by modifying the FE configuration: *

starmgr_grpc_timeout_seconds

(Default is

5

). Increasing this to

10

or

20

can help mitigate transient spikes. Summary: You don't need to panic, but you should monitor the "Vacuum Errors" metric. If it doesn't clear up on its own, investigate the FE load or increase the timeout setting. References • administration/management/FE_configuration.md • faq/shared_data_faq.md • administration/management/monitoring/metrics-shared-data.md • administration/management/logs.md References • administration/management/FE_configuration.md • faq/operation_maintenance_faq.md • administration/management/BE_configuration.md • sql-reference/System_variable.md

@Rocky select count(*) is taking a while on some of my tables, i thought that was metadata? does that mean it hasn't been refreshed or a while?

In a StarRocks shared-data cluster,

SELECT COUNT(_)

is not always a simple metadata lookup. While it _can* be near-instant, several factors can force a much slower execution. Here is why your

COUNT(*)

might be taking a while and how to verify if it’s using metadata: 1. Metadata Optimization vs. Data Scan StarRocks tries to optimize

SELECT COUNT(*)

by using the tablet metadata stored in the FE. However, this optimization is bypassed if: * The query is not a simple `COUNT()`:* Adding any

WHERE

clause or

GROUP BY

forces a scan of the actual data. * Table Type: The optimization is most effective on the Duplicate Key and Primary Key models. On Aggregate or Unique Key models (without Primary Key index), the system may still need to perform some level of background aggregation to provide an accurate count. * The Optimizer chooses a Scan: Sometimes the Cost-Based Optimizer (CBO) decides a scan is more "reliable" if statistics are stale. How to check: Run

EXPLAIN SELECT COUNT(*) FROM your_table;

. * If you see

OlapScanNode

with

isMeta: true

, it is using metadata. * If you see a standard scan without the

isMeta

flag, it is reading files from S3/object storage. 2. The "Cold Cache" Problem (Shared-Data Specific) In a shared-data cluster, even "metadata" operations sometimes require the BEs to fetch tablet headers from remote storage if they aren't in the local Data Cache. * If your "Data Cache Hit Ratio" is low, the BE must wait for object storage (S3/GCS) to return the segment footers to determine the row counts. * This matches the "Deadline Exceeded" errors you saw earlier—if the BE is struggling to fetch tablet info from the FE or storage, the query will hang. 3. High Version Count (Compaction Score) Shared-data tables accumulate "versions" every time you load data. If you have a high volume of small loads and compaction isn't keeping up: * The system has to check many different metadata versions to calculate the final count. * Check your compaction health with:

sql
    SELECT * FROM information_schema.partitions_meta WHERE table_name = 'your_table' ORDER BY MAX_CS DESC;

If

MAX_CS

(Max Compaction Score) is high (e.g., > 100), the metadata overhead for that table is significant. 4. Stale Statistics You mentioned "refreshing." StarRocks periodically collects statistics (including

row_count

) for the CBO. If these are very old, the query planner might make inefficient decisions. * You can check the last time stats were updated:

sql
    SELECT * FROM _statistics_.column_statistics WHERE table_name = 'your_table';

* You can manually trigger a refresh to see if it helps:

sql
    ANALYZE TABLE your_table;

Summary of What to Do: 1. Check

EXPLAIN

: See if

isMeta: true

is present. If it's

false

, a full scan is happening. 2. Check FE Health: Since you previously saw

Deadline Exceeded

errors, your FE might be too busy to serve the metadata requests quickly. 3. Check Compaction: High version counts make even metadata lookups slow. 4. Check Cache: Ensure your

datacache

is enabled and has enough disk space so metadata and headers stay local. References • using_starrocks/Cost_based_optimizer.md • knowledge/admin_and_operation/shared_data_compaction.md • deployment/shared_data/feature-support-shared-data.md • sql-reference/sql-functions/aggregate-functions/count.md • faq/shared_data_faq.md References • faq/shared_data_faq.md