@Rocky I have grafana stood up with the dashboard for shared-data general and shared-data starlet from here https://docs.starrocks.io/docs/administration/management/monitoring/Monitor_and_Alert/#125-configure-dashboard what are some good metrics to monitor for cluster stability? or what are the most important metrics to look at and get right?

For a StarRocks shared-data cluster, monitoring focus shifts from local disk management to the health of the object storage interface (Starlet) and the metadata publishing process. Below are the most critical metrics to monitor for cluster stability, categorized by the dashboards you mentioned. 1. Shared-data General Dashboard This dashboard tracks the coordination between the FE (Frontend) and storage. * Publish Version (Latency & Queued Tasks): This is the most critical metric for data freshness. If "Queued Tasks" is growing, it means the cluster cannot keep up with commits, and your data will become "stale" (queries won't see the latest loads). * Metadata (Put/Get Tablet Metadata Latency): These metrics track how long it takes to interact with the object storage metadata. High spikes here usually indicate performance issues with your object storage (S3/GCS/Azure) or network bottlenecks. * Vacuum: Monitor the success and progress of vacuum tasks. In shared-data, vacuuming is what actually deletes orphaned or old data from object storage to save costs. 2. Shared-data Starlet Dashboard Starlet is the internal filesystem abstraction for object storage. * FSLIB Cache Hit Ratio: In shared-data mode, the local disk is a cache. If this ratio drops significantly, your query performance will degrade as StarRocks has to fetch data directly from object storage (higher latency). * S3 IO Metrics (Latency & IOPS): Specifically look at

fslib s3 single/multi upload latency

. If these values spike, it usually indicates that your object storage provider is throttling you or is experiencing high latency. * FSLIB Async Tasks: This shows the pressure on the background cache loading/writing. A high number of pending async tasks can lead to slow local cache warm-up. 3. General Cluster Health (Essential for Stability) Regardless of the dashboard, these "Red Line" metrics indicate a cluster is about to fail or OOM: * BE Mem (Process Mem): Monitor this against the

mem_limit

(usually 90% of RAM). If BE memory usage exceeds 90%, the BE may trigger OOM-Killer or self-terminate to prevent data corruption. * FE JVM Heap Stat: If this stays above 80%, you will see frequent "Stop-the-World" Garbage Collection, leading to FE timeouts and cluster unreachability. * Compaction Score: While shared-data handles compaction differently, a high compaction score still indicates a bottleneck. If it keeps rising, query performance will deteriorate. * BE CPU Idle: If idle rate is consistently below 10%, the cluster is over-saturated, which often leads to query timeouts and metadata heartbeat failures. Recommended Alert Thresholds Metric | Alert Threshold | Why it matters FE JVM Heap | > 80% | High risk of FE crashes/OOM. BE Mem Usage | > 90% | Impending BE crash or query cancellation. Publish Version Queued | > 50 (or rising) | Data visibility delay; load bottleneck. S3 Error Count | > 0 | Indicates network or permission issues with storage. Disk Avail (Local) | < 20% | Local cache and logs need space; BE may stop if disk is full. For a deep dive into these metrics, refer to the StarRocks Monitoring Metrics Documentation. References • administration/management/monitoring/metrics-shared-data.md • administration/management/monitoring/metrics.md • administration/management/monitoring/Monitor_and_Alert.md • using_starrocks/caching/block_cache.md • knowledge/admin_and_operation/shared_data_compaction.md

@Rocky I see higher values in Query Error than I would suspect, how do I know where they are coming from?

To identify the source of query errors appearing in your Grafana dashboard, you need to correlate the high-level metrics with specific log data or internal metadata. Grafana shows that errors are happening; the following tools tell you who and what is causing them. 1. The FE Audit Log (Most Direct) The

fe/log/fe.audit.log

is the primary source for query attribution. Every query (including failed ones) is logged here by default. * What to look for: Search for entries where

State=ERR

. * Key Fields: * `User`: Which user account ran the query. * `Client`: The IP address of the application or user. * `ErrorCode`: The specific StarRocks error code. * `ErrorMsg`: A descriptive reason for the failure (e.g., syntax error, timeout, or OOM). * `Stmt`: The actual SQL that failed. 2. Query Detail API (Recent Queries) If the errors are happening right now, you can use the Query Detail feature. This caches the most recent queries in FE memory. * Enable it: Ensure

enable_collect_query_detail_info = true

is set in your

fe.conf

. * Access it: You can query the recent failures via the FE HTTP API or, if you have a recent version (3.x+), check if the

AuditLoader

plugin is installed to query this data via SQL. * SQL approach (if AuditLoader is set up):

sql
    SELECT user, clientIp, stmt, errorCode, errorMessage
    FROM starrocks_audit_db*.starrocks_audit_tbl*
    WHERE state = 'ERR'
    ORDER BY timestamp DESC LIMIT 20;

3. System logs (For Internal Errors) If the Audit Log shows generic "Internal Error" messages or RPC timeouts, the root cause is likely deeper in the system: *

fe.log

: Check for

WARNING

or

ERROR

levels. This often captures planning errors, metadata locks, or connection issues with BEs. *

be.INFO.log

/

be.WARN.log

: If a query is "Cancelled" or fails due to resource limits, the BE logs will contain the specific reason, such as: *

Memory limit exceeded

(OOM) *

Table id: xxx is corrupted

*

Timeout

4. Common Sources of "Unexpected" Errors * Resource Groups: If you use Resource Management, queries may be cancelled or rejected if they exceed the group's CPU/Memory limits. * Connectivity/Timeouts: High error counts can sometimes be caused by health checks or monitoring tools using incorrect credentials or hitting

query_timeout

. * Automated Jobs: Check for scheduled

INSERT INTO ... SELECT

or Refresh Materialized View tasks that might be failing in the background. Tip: If you see a specific

QueryId

in your Grafana/Prometheus data, you can grep that ID directly in the

fe.audit.log

to see the full context of that specific failure. References • faq/operation_maintenance_faq.md • administration/management/audit_loader.md • administration/management/logs.md • administration/sql_digest.md

@Rocky jvm heap looks a little high, how can i keep that down?

In StarRocks (especially in shared-data mode), the FE JVM heap mainly stores metadata (tablets, schemas), load task history, and query caches. If your heap usage is high, you can keep it down by tuning these specific areas: 1. Tune Load & Metadata Retention (Quick Wins) The FE keeps a history of load tasks and metadata in memory. Reducing these limits can significantly lower heap pressure. *

stream_load_task_keep_max_num

: Controls how many Stream Load records are kept in memory. Reducing this from the default

1000

to

500

or

300

will free up space. *

label_keep_max_second

: Determines how long load labels are kept. Reducing this (e.g., from 3 days to 1 day) reduces the volume of transaction metadata in the heap. *

history_job_keep_max_second

: Reduces the retention of completed job metadata (Export, Spark Load, etc.). 2. External Catalog Metadata Caching If you use External Catalogs (Hive, Iceberg, JDBC), the FE caches file list and schema information. * Lower Cache TTL: Reduce the expiration time for catalog metadata to force more frequent eviction. * Disable Default Caching: For JDBC catalogs, ensure

jdbc_meta_default_cache_enable

is

false

if you don't need persistent metadata caching. 3. Shared-Data Specific: Metadata Pressure In shared-data mode, an excessive number of shards/tablets can overwhelm the FE during cleaning cycles. *

meta_sync_force_delete_shard_meta

: If you have a massive number of dropped tables/shards pending cleanup, setting this to

true

allows FE to delete metadata without waiting for the slow object storage deletion process, which can prevent OOMs during cleanup (available in v3.2.10+). 4. JVM & GC Configuration Ensure your JVM settings are optimized for the workload. * Sizing: Ensure

-Xms

and

-Xmx

are set to the same value in

fe.conf

. StarRocks recommends 16GB for <1M tablets and 32GB for 1M–2M tablets. * G1GC: Ensure you are using the G1 Garbage Collector (usually the default in modern StarRocks installs). It handles large heaps more gracefully than CMS. *

MALLOC_ARENA_MAX=1

: Add this to your environment variables. While it technically targets off-heap memory, it prevents the glibc allocator from creating too many arenas, which can indirectly cause the process to be killed by the OS even if the heap looks "okay." 5. Managing Tablets (Long-term Strategy) The single biggest driver of FE heap usage is the total number of Tablets. * Consolidate Partitions: If you have thousands of empty or tiny partitions, delete them or use dynamic partitioning to expire old data. * Bucket Tuning: Increase your bucket size (target 100MB–1GB per bucket) to reduce the total tablet count. How to diagnose "What is in my Heap?" If the usage stays high after tuning, use these built-in tools: * Memory Usage Tracker (v3.3.7+): Check

fe/log/fe.log

for regular "MemoryUsageTracker" entries. It breaks down which module (e.g., LoadManager, Catalog, TabletManager) is consuming the most memory. * Memory Profiles (v3.3.6+): Check

fe/log/proc_profile/

. The FE now automatically dumps

.tgz

flame graphs when memory surges. Open these in a browser to see the exact stack trace allocating the most memory. * Manual Check: Run

jmap -histo:live <pid> | head -n 20

to see which Java object types are dominating the heap. References • knowledge/trouble_shooting/fe_memory_problems.md • faq/fe_mem_faq.md • administration/management/FE_configuration.md • faq/Others.md References • administration/management/FE_configuration.md

@Rocky is there an easy way to get tablet count?

There are three "easy" ways to get the tablet count, depending on whether you want a quick summary or a detailed SQL result. 1. The Quickest Summary (Total Count) Run the following command in your MySQL client. It provides a cluster-wide summary, including a Total row at the bottom:

sql
SHOW PROC '/statistic';

* TabletNum: The total number of tablets across the cluster (the bottom "Total" row is your global count). * UnhealthyTabletNum: Tells you if any tablets are currently in a bad state. 2. The SQL Way (Aggregated by Table/Database) If you are on StarRocks 3.1.9 or 3.2.4+, you can use the system metadata tables. This is best if you want to find which specific table is "bloating" your tablet count:

sql
SELECT 
    db_name, 
    table_name, 
    SUM(buckets) as total_tablets
FROM information_schema.partitions_meta
GROUP BY db_name, table_name
ORDER BY total_tablets DESC;

Note: In shared-data mode, the

buckets

count is equivalent to the

tablet

count because replication is handled by the storage layer (S3/GCS). 3. Check Distribution Across BEs If you want to see how tablets are distributed among your BE nodes (to check for hotspots), use:

sql
SHOW PROC '/backends';

Look at the TabletNum column for each BE. Why does this matter for stability? As you noted with the JVM heap usage, the FE has to track every tablet in the cluster. * Rule of Thumb: A general recommendation is to keep the total tablet count under 1 million per 16GB of FE RAM. * The "Small Tablet" Problem: If your tablet count is high but your data volume is low, you likely have too many partitions or your bucket count is set too high. Increasing your bucket size (aim for 100MB–1GB per tablet) can significantly improve FE stability and query planning performance. References • sql-reference/sql-statements/cluster-management/nodes_processes/SHOW_PROC.md • knowledge/admin_and_operation/shared_data_compaction.md • administration/management/monitoring/metrics.md • faq/operation_maintenance_faq.md References • sql-reference/sql-statements/cluster-management/nodes_processes/SHOW_PROC.md

@Rocky what is the downside to decreasing

stream_load_task_keep_max_num

Decreasing

stream_load_task_keep_max_num

is an effective way to lower FE heap usage, but the primary downside is the loss of historical visibility for your data ingestion. Specifically, if you set this value too low: 1. Harder Debugging and Auditing When a Stream Load fails, you typically check its status via

SHOW LOAD

or the FE's internal tracking. If the task record is purged from memory because the limit was reached, you will no longer be able to see: * Why a specific job failed (ErrorMsg). * How many rows were actually loaded. * The performance metrics (latency, data size) of that specific load. 2. Issues with Label Checking (Deduplication) StarRocks uses labels to prevent double-loading the same data. * If your application finishes a load and then immediately tries to query the status of that label to verify success, it might receive a "Label Not Found" error if the cleanup was too aggressive. * While this doesn't break the data consistency itself, it can confuse orchestration tools or retry logic in your ingestion pipeline. 3. Rapid Turnover in High-Frequency Loading If you are performing hundreds of small Stream Loads per second, a default limit of

1000

already represents only a few seconds of history. Decreasing it further in a high-frequency environment means records might exist for only a fraction of a second before being evicted. Summary Recommendation * Don't go too low: If you have high load frequency, try decreasing it gradually (e.g., from 1000 to 500) rather than a drastic drop. * Check the Time Limit too: Ensure

stream_load_task_keep_max_second

(default 3 days) isn't also unnecessarily high, as both parameters work together to trigger cleanup. * Alternative: Instead of just decreasing the count, try to batch your Stream Loads into larger chunks. Fewer, larger loads create significantly less metadata pressure on the FE than many tiny loads. References • administration/management/FE_configuration.md • faq/fe_mem_faq.md • administration/management/FE_configuration.md References • administration/management/FE_configuration.md • administration/management/BE_configuration.md • loading/Kafka-connector-starrocks.md

@Rocky is there a way to look at my stream load history to inspect what i have and how old it is?

Yes, there are two primary ways to inspect your stream load history via SQL. Depending on your StarRocks version, one may be more convenient than the other. 1. The Modern Way:

information_schema.loads

(Recommended) Starting from v3.1, StarRocks provides a unified view for all load types. This is the easiest way to see your history in a human-readable format.

sql
SELECT 
    label, 
    db_name, 
    table_name, 
    state, 
    create_time, 
    load_start_time, 
    load_finish_time,
    TIMESTAMPDIFF(SECOND, create_time, load_finish_time) as duration_sec
FROM information_schema.loads
WHERE type = 'STREAM'
ORDER BY create_time DESC
LIMIT 100;

* Why use this: It uses standard timestamp formats (

yyyy-MM-dd HH:mm:ss

) and includes all load types if you want to compare Stream Load vs. Broker Load. 2. The Detailed Way:

information_schema.stream_loads

This view is specific to Stream Loads and provides more granular timing (in milliseconds), which is useful for debugging high-frequency ingestion.

sql
SELECT 
    label, 
    table_name, 
    state, 
    FROM_UNIXTIME(create_time_ms / 1000) as created_at,
    (end_time_ms - create_time_ms) as total_latency_ms,
    num_rows_normal,
    error_msg
FROM information_schema.stream_loads
ORDER BY create_time_ms DESC;

* Why use this: It provides internal metrics like

begin_txn_ms

and

receive_data_time_ms

(found in the

runtime_details

column in some versions) which help pinpoint if the slowness is in the network or the transaction commit. 3. Long-term History:

statistics.loads_history

By default, the views above only show what is currently in the FE's memory (controlled by the

stream_load_task_keep_max_num

you mentioned). If you need to see history from weeks ago:

sql
SELECT * FROM statistics.loads_history 
WHERE type = 'STREAM' 
ORDER BY create_time DESC;

* Note: This view stores records for up to 3 months by default and is persisted to disk, so it won't impact your FE heap as much as the in-memory views. Summary of Timing Columns: *

create_time

: When the FE first received the load request. *

load_start_time

: When the actual data transmission/processing began. *

load_finish_time

/

end_time_ms

: When the transaction was fully committed and data became visible. If you find that your history is very short (e.g., only a few minutes of logs), it's a sign that your

stream_load_task_keep_max_num

is being hit frequently due to high-frequency small-batch loading. References • sql-reference/information_schema/loads.md • sql-reference/information_schema/stream_loads.md • loading/loading_introduction/troubleshooting_loading.md References • sql-reference/information_schema/stream_loads.md • sql-reference/information_schema/loads.md