http://starrocks.io logo
Join Slack
Powered by
<@U0A71G31CDV> how to check if statictics collecti...
# questions-and-troubleshooting
z
@Rocky how to check if statictics collection is stuck and is in queue? in shared cluster
r
In a StarRocks shared cluster (shared-data), you can check the status and queue of statistics collection using the following methods: 1. Check the Status of Collection Tasks The most direct way to see if tasks are queued or running is using the 
SHOW ANALYZE
commands: * For all tasks (automatic and manual):
Copy code
sql
    SHOW ANALYZE STATUS [WHERE Table = 'your_table_name'];
* PENDING: The task is in the queue waiting for resources. * RUNNING: The task is currently executing on the CN (Compute Nodes). * FAILED: The task stopped; check the Reason column for the error (e.g., timeout or resource issues). * For custom/manual jobs:
Copy code
sql
    SHOW ANALYZE JOB;
2. Identify if it's "Stuck" A task might be considered "stuck" if it stays in 
PENDING
or 
RUNNING
for an unexpectedly long time. * Queued (PENDING): If many tasks are in 
PENDING
, the cluster might be hitting the concurrency limit. You can check the FE configuration 
statistic_collect_parallel
(default is 
1
), which controls how many tasks run concurrently. * Running (RUNNING): If a task is 
RUNNING
but not finishing, check the cluster's resource usage (CPU/Memory) on your CN nodes. Statistics collection is essentially a query; you can see if it appears in 
SHOW PROCESSLIST
or 
SHOW RUNNING QUERIES
. 3. Check Statistics Health To see if statistics are outdated or not being triggered, use:
Copy code
sql
SHOW STATS META [WHERE Table = 'your_table_name'];
This shows the Healthy column (a value closer to 100 is better) and the UpdateTime. If the table has been updated recently but 
UpdateTime
is old, a task may be stuck or haven't been triggered yet. 4. Shared Cluster Specifics In a shared-data cluster, statistics collection tasks are dispatched to CN nodes. * CN Load: If your CN nodes are under heavy load or have insufficient memory, statistics collection (especially 
FULL
collection on large tables) may be slow or fail. * Internal Tables: You can query the raw statistics directly to see what has been successfully recorded:
Copy code
sql
    SELECT * FROM _statistics_.column_statistics WHERE table_name = 'your_table_name';
Troubleshooting Stuck Tasks * Increase Parallelism: If the queue is too long, you can try increasing the parallelism temporarily (via FE config):
Copy code
sql
    -- Example to check current FE configs
    ADMIN SHOW FRONTEND CONFIG LIKE '%statistic%';
* Manual Trigger: If an automatic task is stuck, you can try to manually trigger a sampled collection to reduce resource load:
Copy code
sql
    ANALYZE TABLE <table_name> WITH SAMPLE;
* FE Logs: Check the 
fe.log
for any errors containing 
Analyze
or 
Statistic
to see if the FE is failing to schedule the tasks. References • sql-reference/information_schema/statistics.md • using_starrocks/Cost_based_optimizer.md • sql-reference/information_schema/tables.md
z
@Rocky its about external table statistics and they are stuck in a queue and what is the solution?
r
If your external table statistics are stuck in a queue (
PENDING
status), it is usually because they have hit the default concurrency limits or because external tables require more resources (as they only support Full Collection, not sampled). Here is how to check the status and the solutions to clear the queue: 1. Check the Queue Status First, verify exactly how many tasks are queued: * For all tasks: 
SHOW ANALYZE STATUS;
* Look for tasks where the state is 
PENDING
. * For automatic jobs: 
SHOW ANALYZE JOB;
* For Query-Triggered tasks (v3.4+): Since these are background tasks triggered by queries, they are governed by specific FE configurations (see below). 2. Solutions to Clear the Queue A. Increase Global Collection Parallelism By default, StarRocks limits how many statistics tasks can run in parallel to avoid impacting query performance. You can increase this globally:
Copy code
sql
-- Increase the number of concurrent statistics tasks
SET GLOBAL statistic_collect_parallel = 4; -- Default is 1
B. Increase Query-Triggered Concurrency (v3.4+) If you are on version 3.4 or later, external table statistics are often "Query-Triggered." These have their own dedicated limits in the FE configuration: * `connector_table_query_trigger_analyze_max_running_task_num`: Controls how many external table tasks run at once (Default: 2). * `connector_table_query_trigger_analyze_max_pending_task_num`: Controls the queue size (Default: 100). To increase these, run this on the FE:
Copy code
sql
ADMIN SET FRONTEND CONFIG ("connector_table_query_trigger_analyze_max_running_task_num" = "5");
C. Manually Trigger with High Priority If a specific large table is stuck, you can try running a Synchronous manual analyze task in your current session. This bypasses the background scheduler's queue limits for that specific command:
Copy code
sql
ANALYZE TABLE <catalog>.<db>.<table> WITH SYNC MODE;
Note: External tables do NOT support 
WITH SAMPLE
. They must perform a 
FULL
scan, so ensure your CN nodes have enough resource overhead. D. Check for "Stuck" Tasks in FE Logs If the queue is not moving even after increasing parallelism, check the 
fe.log
on the leader FE node. Search for "Analyze" or "Statistic": * Reason: Look for errors like 
Precondition failed
or connection timeouts to the external catalog (Hive/Iceberg). * Resource limits: If the CN (Compute Nodes) are under extreme memory pressure, the FE may delay scheduling new tasks. Summary Checklist 1. Check Status: 
SHOW ANALYZE STATUS
(identify 
PENDING
). 2. Increase Parallelism: 
SET GLOBAL statistic_collect_parallel = 5;
. 3. Adjust FE Config: Increase 
connector_table_query_trigger_analyze_max_running_task_num
. 4. Verify Catalog Connectivity: Ensure the cluster can still reach the Hive Metastore or S3/HDFS. References • using_starrocks/Cost_based_optimizer.md
z
@Rocky 
SHOW ANALYZE STATUS
there is no pending here
r
If 
SHOW ANALYZE STATUS
does not show any 
PENDING
tasks, but you suspect external table statistics are not being collected or are "stuck," the issue likely lies in how the task is triggered or queue limits that prevent the task from even being registered in the status table. Here is how to troubleshoot and solve this for external tables: 1. Check if the Job Exists For external tables (Hive/Iceberg/etc.), StarRocks does not automatically collect statistics in the background unless: * You have manually created an Analyze Job. * Query-Triggered Analyze is enabled (v3.4+). Run this to see if any background jobs are actually defined:
Copy code
sql
SHOW ANALYZE JOB;
If this is empty, no automatic background collection is happening. You must create a job first:
Copy code
sql
CREATE ANALYZE JOB <job_name> FOR TABLE <catalog>.<db>.<table>;
2. Query-Triggered Analyze (v3.4+) If you are relying on StarRocks to trigger stats automatically when you run queries, these tasks are placed in an internal FE queue. If the queue is full, new tasks are silently discarded and will not appear in 
SHOW ANALYZE STATUS
. * Check the current limits:
Copy code
sql
    ADMIN SHOW FRONTEND CONFIG LIKE 'connector_table_query_trigger%';
* Check if the queue is full: Look in your 
fe.log
for messages like: 
"PendingTaskQueue is full, discard the task"
* Solution: Increase the queue size and the number of running tasks:
Copy code
sql
    ADMIN SET FRONTEND CONFIG ("connector_table_query_trigger_analyze_max_pending_task_num" = "500");
    ADMIN SET FRONTEND CONFIG ("connector_table_query_trigger_analyze_max_running_task_num" = "10");
3. Verify Metadata Health If you think stats are stuck because queries are slow, check the health of the statistics:
Copy code
sql
SHOW STATS META WHERE Table = 'your_table_name';
* If Healthy is low (e.g., < 50) and no task is in 
SHOW ANALYZE STATUS
, the scheduler hasn't decided to trigger a collection yet. * Check the 
UpdateTime
. If it's very old, a manual trigger is needed. 4. Force a Synchronous Collection (The "Clear" Solution) If you need statistics immediately and the background queue isn't picking it up, run the collection in Synchronous Mode. This bypasses the background scheduler and runs the task in your current session:
Copy code
sql
-- This will run immediately and block until finished
ANALYZE TABLE <catalog>.<db>.<table> WITH SYNC MODE;
Note: For external tables, only 
FULL
collection is supported. Ensure your CN nodes have enough memory as this will perform a scan of the external table. 5. Check the Internal Stats Table To see exactly what has been recorded for your external table (independent of the status UI), query the internal statistics table:
Copy code
sql
SELECT * FROM _statistics_.external_column_statistics 
WHERE table_name = 'your_table_name';
Summary of common "stuck" reasons: 1. Queue Discard: The 
PendingTaskQueue
reached 
connector_table_query_trigger_analyze_max_pending_task_num
and ignored the request. 2. No Job: No 
ANALYZE JOB
was created for the external table. 3. Hudi Tables: StarRocks cannot perceive data changes in Hudi; it only collects periodically based on 
statistic_auto_collect_small_table_interval
. References • administration/management/FE_configuration.md • using_starrocks/Cost_based_optimizer.md • sql-reference/System_variable.md
z
@Rocky so in case of iceberg tables its meta statistics collection is fast because it has manifest from where starrocks can take data and what starrocks internal format? .dat? it can be slow?
r
For Iceberg tables, there is a distinction between metadata-based planning (which is fast) and CBO statistics collection (which can be slow). 1. Metadata Planning vs. Statistics Collection * Metadata Planning (Fast): StarRocks can use Iceberg's manifest files to get basic info (like row counts and file paths) for query planning. This is fast because it only reads small Avro/metadata files. StarRocks uses a "Distributed Metadata Plan" to parallelize this across CN/BE nodes. * Statistics Collection (
ANALYZE
) (Slow): To get the detailed statistics needed for the Cost-Based Optimizer (CBO)—such as NDV (Number of Distinct Values), Histograms, or *Null Counts*—StarRocks currently requires a Full Collection for external tables. * Why it's slow: Unlike internal tables, StarRocks does not support Sampled Collection for Iceberg. It must perform a full scan of the data files to calculate accurate NDV and histograms. Even though manifests have some column-level min/max, they are often insufficient for the high-quality statistics StarRocks' CBO requires. 2. Internal Format and Storage (.dat) You mentioned 
.dat
files; in StarRocks, this typically refers to the Segment Files used by the internal storage engine. * Storage Location: When you run 
ANALYZE
, the results are stored in internal tables within the 
_statistics_
database (e.g., 
_statistics_.external_column_statistics
). * Format: These internal tables are stored in StarRocks' own columnar format. In the shared-data (shared cluster) architecture, these are eventually uploaded to your object storage (S3/GCS/HDFS) as segment files. * Performance Bottleneck: The "slowness" is rarely the writing of these statistics files (which are very small). The bottleneck is the IO and CPU cost of scanning the entire Iceberg data set to compute the statistics in the first place. 3. Why it might be "Stuck" or Slow in your Shared Cluster If the statistics collection seems slow or stuck even if no "PENDING" tasks show: 1. Full Scan Overhead: For a multi-terabyte Iceberg table, a 
FULL ANALYZE
is essentially a massive aggregation query over the entire table. 2. Resource Competition: In a shared cluster, these tasks run on CN nodes. If your CNs are busy with user queries, the background 
ANALYZE
tasks (which have lower priority) will take much longer. 3. IO Throttling: Since it's a shared cluster, the speed is limited by the network bandwidth between your CN nodes and the object storage where the Iceberg data resides. Summary Recommendation * Use Manual Analyze for Large Tables: If automatic collection is too slow, trigger it manually for specific columns only:
Copy code
sql
    ANALYZE TABLE <catalog>.<db>.<table> (column1, column2);
* Check `fe.log`: If you suspect the task is being discarded before it even hits the queue, look for "discard the task" or "PendingTaskQueue is full" in the FE logs. References • data_source/catalog/iceberg/iceberg_catalog.md • using_starrocks/Cost_based_optimizer.md • data_source/feature-support-data-lake-analytics.md References • using_starrocks/Cost_based_optimizer.md • data_source/catalog/iceberg/iceberg_catalog.md
z
@Rocky no what I meant is if I also have starrocks internal format tables they even dont have some manifest jsons like iceberg so they have no meta separately and meta-data planning cant be involved?
r
You are correct that StarRocks internal tables do not have "manifest JSON files" in the way Iceberg does, but they actually have a more deeply integrated metadata system that allows for even faster planning. Here is how the "meta-data planning" works for internal tables compared to Iceberg: 1. Where the Metadata Lives * Iceberg: The "source of truth" is on your object storage (S3/HDFS) in Avro/JSON manifest files. StarRocks must fetch, decompress, and parse these files (metadata planning) just to know which data files to read. This is why "Distributed Planning" was added for Iceberg—to speed up this external file-reading step. * Internal Tables: The metadata (partition ranges, tablet locations, row counts, and version info) is stored directly in the StarRocks FE (Frontend). It is kept in memory and persisted in the FE's 
image
and 
edit logs
. * Result: The planning is almost instantaneous because the FE doesn't have to "fetch" metadata from a remote disk; it already owns it. 2. The Internal "Manifest" (Segments) While there isn't a separate JSON file, StarRocks segment files (the 
.dat
files you mentioned) are self-describing. * Segment Footer: Every segment file contains a footer that acts as a local "manifest." It stores Zone Maps (min/max values, null counts) for every column and every data page (64KB blocks). * Read-time Pruning: During a query, the BE (Backend) reads the footer first. If the 
min/max
in the footer shows that the data doesn't match your 
WHERE
clause, the BE skips the entire file without reading the actual data. This is functionally equivalent to Iceberg's manifest pruning but happens at the storage engine level. 3. CBO Statistics (The 
.dat
issue) The "slowness" you might be thinking of regarding internal statistics is the CBO (Cost-Based Optimizer) stats (like NDV/Histograms). * External Tables: StarRocks often has to scan the entire table to calculate NDV because it can't trust the external metadata to be 100% accurate or detailed enough. * Internal Tables: StarRocks supports Sampled Statistics. It can read just a small percentage of your 
.dat
files to estimate the distribution of data. This makes 
ANALYZE
for internal tables much faster than for external tables. Summary Comparison Feature | Iceberg External | StarRocks Internal Meta Storage | Remote Files (S3/HDFS) | FE Memory / Local Disk Planning Cost | High (must parse manifests) | Near Zero (FE knows all) Data Skipping | Based on Manifests | Based on Segment Zone-maps Stats Collection | Full Scan (usually) | Sampled Scan (Fast) Conclusion: StarRocks internal tables don't need manifest files because the FE itself acts as the manifest manager. It is actually faster than Iceberg because there is no "metadata I/O" required during the planning phase of a query. References • table_design/table_types/primary_key_table.md • best_practices/table_clustering.md • using_starrocks/Cost_based_optimizer.md References • best_practices/table_clustering.md • table_design/table_types/primary_key_table.md • data_source/catalog/iceberg/iceberg_meta_table.md • table_design/table_design.md • knowledge/admin_and_operation/shared_data_compaction.md
z
@Rocky so internal tables automated analyze job and query triggered analyze job share the same queue right?
r
The short answer is no, they do not share the same queue limits, even though they are managed by the same background system (the FE's 
AnalyzeManager
) and both appear in 
SHOW ANALYZE STATUS
. StarRocks treats Internal Table Auto-Analyze and External Table Query-Triggered Analyze as two different categories with separate scheduling and capacity limits. 1. Separate Queue Limits While both types of tasks are eventually executed by the FE's background threads, they have different "waiting rooms": * Query-Triggered (External) Queue: This has a very specific and strict limit. If more than 100 tasks (by default) are waiting, new query-triggered tasks are silently dropped. _ _Config:* 
connector_table_query_trigger_analyze_max_pending_task_num
(Default: 100) _ _Running Limit:* 
connector_table_query_trigger_analyze_max_running_task_num
(Default: 2) * Automated/Scheduled (Internal) Queue: These tasks are triggered by a health check that runs every 5 minutes (
statistic_collect_interval_sec
). They do not follow the "connector" limits. Instead, they are governed by general FE background task limits and the health threshold of the table. 2. Why you see no "PENDING" for External Tables The most common reason for 
SHOW ANALYZE STATUS
showing no 
PENDING
tasks for an Iceberg/External table is that the External Queue is full. * When a query triggers an analyze task, if the 
PendingTaskQueue
already has 100 tasks, the system does not queue the 101st task. It simply ignores the request. * Because the task was never accepted into the queue, it never gets a status record, so it doesn't show up in 
SHOW ANALYZE STATUS
. 3. Shared Resource: The Backend (BE/CN) While the FE queues are managed separately, they share the same execution resources on your cluster. * Both types of tasks consume CPU and IO on your CN/BE nodes. * You can control the overall parallelism of how many columns are analyzed at once across the cluster using the session variable:
Copy code
sql
    SET GLOBAL statistic_collect_parallel = 1; -- Increase to speed up, decrease to save resources
Summary Comparison Feature | Internal Auto-Analyze | External Query-Triggered Trigger | Data change health check (5 min) | Running a query on the table Queue Limit | Managed by FE Memory/Threads | 
connector_table_query_trigger_analyze_max_pending_task_num
Max Running | Internal FE scheduling | 
connector_table_query_trigger_analyze_max_running_task_num
Behavior when full | Scheduled for next check | Task is discarded/dropped Recommendation: If you have many Iceberg tables and 
SHOW ANALYZE STATUS
is empty, increase the connector-specific limits so they actually make it into the queue:
Copy code
sql
ADMIN SET FRONTEND CONFIG ("connector_table_query_trigger_analyze_max_pending_task_num" = "1000");
ADMIN SET FRONTEND CONFIG ("connector_table_query_trigger_analyze_max_running_task_num" = "5");
References • using_starrocks/Cost_based_optimizer.md • administration/management/FE_configuration.md • sql-reference/sql-statements/cbo_stats/SHOW_ANALYZE_STATUS.md References • using_starrocks/Cost_based_optimizer.md • administration/management/monitoring/metrics.md • administration/management/FE_configuration.md • administration/management/BE_configuration.md
z
@Rocky but internal table statistics update at least is run by root user and it resource group has limits what is shared with external statistics run
r
Yes, you are exactly right. Because StarRocks' background maintenance tasks (like statistics collection) are typically executed using the system identity—often associated with the *`root` user*—they are subject to any Resource Group limits you have placed on that user. Here is how the interaction works for internal and external statistics: 1. Shared Identity (The "root" User) Both internal auto-analyze and external query-triggered analyze are initiated by the FE background scheduler. * When these tasks are sent to the BE/CN for execution, they carry the user context of 
root
(or the internal system user). * If you have created a Resource Group with a classifier like 
TO (user='root')
, all statistics tasks (internal or external) will be funneled into that group. 2. Shared Resource Pool (The Resource Group) Once inside that resource group, both types of tasks share the same: * 
cpu_weight
/ 
exclusive_cpu_cores
: They compete for the same CPU slices. * 
mem_limit
: If the total memory used by internal stats and external stats exceeds the group's limit, the system may throttle or fail tasks. * 
concurrency_limit
: If your 
root
resource group has a 
concurrency_limit = 5
, then at most 5 columns (or table tasks) across the entire cluster can be analyzed at once, regardless of whether they are internal or external. 3. Difference in "Gatekeeping" (FE Queues) While they share the same execution resources on the BE, they have different entry gates on the FE: * External (Iceberg/Hive): Controlled by 
connector_table_query_trigger_analyze_max_running_task_num
(Default: 2). Even if your Resource Group has room for 10 tasks, the FE will only send 2 external analyze tasks to the BE at a time. * Internal Table: Controlled by internal FE scheduling logic and 
statistic_collect_parallel
(Default: 1). Example Scenario If you have a resource group for 
root
with `concurrency_limit = 2`: 1. An internal auto-analyze task starts for a large local table (1 slot used). 2. A query triggers an analyze on an Iceberg table. It starts (1 slot used). 3. Both slots are now full. 4. If another local table needs auto-analyze or another external query triggers a task, they will both be blocked/queued at the resource group level, even if the FE configuration (
connector_table_...
) would have allowed more. Summary * Do they share limits? Yes, they share the BE Resource Group limits if they match the same classifier (usually 
user='root'
). * Can it be slow? Yes. If the 
root
resource group is too restrictive, external Iceberg statistics will wait behind internal table updates, and vice-versa. * Recommendation: If your external statistics are failing or slow, check the 
ResourceGroup
column in 
fe.audit.log
for the analyze queries (you may need to enable 
enable_statistics_collect_profile = true
to see more detail). If they are hitting a group with a low 
cpu_weight
or 
concurrency_limit
, you should increase those specifically for the maintenance user. References • administration/management/resource_management/resource_group.md • using_starrocks/Cost_based_optimizer.md • best_practices/audit_log_resource_group.md References • sql-reference/sql-statements/cbo_stats/ANALYZE_TABLE.md • administration/management/FE_configuration.md • using_starrocks/Cost_based_optimizer.md • administration/management/monitoring/metrics.md
z
@Rocky statistics collection is done by CNs or FEs?
r
Statistics collection is a joint effort between the FE (Frontend) and the BE/CN (Backend/Compute Node), but the actual data-heavy "collection" (scanning and calculating) is done by the BEs or CNs. Here is the breakdown of who does what: 1. The FE (The Coordinator) The FE acts as the brain and manager for statistics. It does not scan the data itself. * Triggering: The FE decides when a table is "unhealthy" and needs an update, or it receives your manual 
ANALYZE
command. * Planning: The FE generates an internal execution plan (essentially a hidden 
SELECT
query) to calculate counts, distinct values (NDV), and histograms. * Scheduling: It places the task in a queue (
PendingTaskQueue
) and manages the concurrency. * Caching: Once the results are ready, the FE caches a copy of the statistics in its memory so the Optimizer can access them instantly during query planning. 2. The BE/CN (The Worker) The BEs (in shared-nothing) or CNs (in shared-data) are the muscles. They do the actual work. * Data Scanning: The BEs/CNs read the data files (from local disk or remote object storage like S3). * Computation: They perform the heavy lifting of calculating 
min
, 
max
, 
null_count
, and the 
HyperLogLog
(HLL) for cardinality estimation. * Persistence: The computed statistics are written into internal StarRocks system tables (e.g., 
_statistics_.column_statistics
). Since these are StarRocks tables, the data is physically stored on the BEs. Summary * Collection (Scanning/Calculating): Done by BEs or CNs. * Management & Storage of Metadata: Done by the FE. * Persisted Storage of Stats Data: Done on the BEs (inside system tables). In short: The FE tells the BE/CN what to do; the BE/CN reads the data and calculates the numbers; the FE then remembers those numbers to make your queries faster. References • using_starrocks/Cost_based_optimizer.md • introduction/Architecture.md • administration/management/resource_management/resource_group.md References • introduction/Architecture.md • using_starrocks/Cost_based_optimizer.md • best_practices/query_tuning/query_planning.md
Open in Slack
PreviousNext
Powered by