@Rocky When I change my cluster topology for shared-data, what is the quickest way to rebalance the data from S3 onto local cache?

When you change the topology of a StarRocks shared-data cluster (e.g., adding or removing Compute Nodes), the tablets are reassigned to different nodes. Since the underlying data resides in S3, the "rebalancing" process primarily involves populating the local disk cache on the new nodes to avoid performance degradation. The quickest and most effective way to handle this depends on your StarRocks version: 1. Enable Cache Sharing (Best for minimizing S3 traffic) Starting from StarRocks v3.1+, you can enable Cache Sharing. Instead of every node fetching data from S3 after a topology change, a node that misses a local cache can fetch it directly from other nodes in the cluster over the network. This is significantly faster and cheaper than re-fetching from S3. * FE Configuration: Set

enable_trace_historical_node = true

. * System Variable: Run

SET GLOBAL enable_datacache_sharing = true;

. 2. Proactive Warmup via

CACHE SELECT

(v3.3+) If you want to manually trigger the data movement from S3 to the local cache after a topology change, use the Data Cache Warmup feature introduced in v3.3. This allows you to proactively fetch data into the cache using a SQL command rather than waiting for user queries to populate it. * Command:

sql
    CACHE SELECT * FROM your_table WHERE your_filter;

* Background Task: You can submit this as a background task to avoid blocking your session:

sql
    SUBMIT TASK warmup_task AS CACHE SELECT * FROM your_table;

3. Automatic Tablet Balancing Ensure the FE is configured to automatically rebalance the tablet assignments across your new topology. In v3.3.4 and later, the following parameter is enabled by default: *

lake_enable_balance_tablets_between_workers = true

(FE configuration) Summary of Best Practice 1. Before/During scaling: Ensure Cache Sharing is enabled. This allows new nodes to "borrow" cache from the old nodes immediately. 2. After scaling: Execute

CACHE SELECT

on your most frequently accessed tables or partitions to ensure the local disks are fully populated without waiting for "cold" starts on user queries. References • deployment/shared_data/feature-support-shared-data.md • data_source/data_cache.md • using_starrocks/caching/block_cache.md • deployment/shared_data/s3.md

@Rocky How to track task progress

To track task progress in StarRocks, the method depends on the type of task you are running. Here is a summary of the most common commands and system views: 1. Data Ingestion (Loading) * Routine Load (Kafka/Pulsar): * To see the overall job status and progress (offsets):

sql
        SHOW ROUTINE LOAD FOR <job_name>;

* To see individual running tasks within that job:

sql
        SHOW ROUTINE LOAD TASK WHERE JobName = '<job_name>';

* Broker Load, Spark Load, and INSERT INTO: * Check the

Progress

column (0-100%):

sql
        SHOW LOAD [FROM <db_name>] WHERE Label = '<label_name>';

2. Asynchronous Tasks & Materialized Views For tasks submitted via

SUBMIT TASK

(like manual cache warmups) or asynchronous Materialized View (MV) refreshes, use the Information Schema: * Find the Task Name:

sql
    SELECT * FROM information_schema.tasks ORDER BY CREATE_TIME DESC;

* Check Execution Progress/Status:

sql
    SELECT * FROM information_schema.task_runs
    WHERE task_name = '<task_name>'
    ORDER BY CREATE_TIME DESC;

Look at the

STATE

and

PROGRESS

(e.g., 50%) columns. 3. Table Alterations & Schema Changes If you are adding columns, creating rollups, or optimizing table bucket structures: * Schema Changes:

SHOW ALTER TABLE COLUMN;

* Rollup/Synchronous MV creation:

SHOW ALTER TABLE ROLLUP;

or

SHOW ALTER MATERIALIZED VIEW;

* Bucket Optimization:

SHOW ALTER TABLE OPTIMIZE;

4. Running Queries To track active SQL queries and see if they are pending in a queue: * Current Processlist:

SHOW PROCESSLIST;

* Queued/Running Info (v3.1.4+):

SHOW RUNNING QUERIES;

5. Background Internal Tasks (Shared-Data) For shared-data clusters tracking compaction or data movement: * Compaction Status:

SHOW PROC '/compactions';

* Detailed Tablet Compaction: Query

information_schema.be_cloud_native_compactions

. References • sql-reference/sql-statements/loading_unloading/SHOW_LOAD.md • sql-reference/sql-statements/table_bucket_part_index/SHOW_ALTER.md • using_starrocks/async_mv/Materialized_view.md • administration/management/resource_management/query_queues.md References • sql-reference/sql-statements/table_bucket_part_index/SHOW_ALTER.md • using_starrocks/Materialized_view-single_table.md • faq/Sql_faq.md • knowledge/admin_and_operation/shared_data_compaction.md

@Rocky MySQL [events]> SELECT * FROM information_schema.task_runs WHERE task_name = 'warmup_task' ORDER BY CREATE_TIME DESC; +--------------------------------------+-------------+---------------------+---------------------+---------+-----------------+----------+----------------------------+---------------------+------------+---------------+----------+---------------+-----------------------------------+--------+---------------------+ | QUERY_ID | TASK_NAME | CREATE_TIME | FINISH_TIME | STATE | CATALOG | DATABASE | DEFINITION | EXPIRE_TIME | ERROR_CODE | ERROR_MESSAGE | PROGRESS | EXTRA_MESSAGE | PROPERTIES | JOB_ID | PROCESS_TIME | +--------------------------------------+-------------+---------------------+---------------------+---------+-----------------+----------+----------------------------+---------------------+------------+---------------+----------+---------------+-----------------------------------+--------+---------------------+ | 019f28b4-86c6-7363-bcfb-05755262f3c5 | warmup_task | 2026-07-03 155907 | 1970-01-01 000000 | RUNNING | default_catalog | events | CACHE SELECT * FROM events | 2026-07-10 155907 | 0 | NULL | 0% | NULL | {"warehouse":"default_warehouse"} | NULL | 2026-07-03 155907 | +--------------------------------------+-------------+---------------------+---------------------+---------+-----------------+----------+----------------------------+---------------------+------------+---------------+----------+---------------+-----------------------------------+--------+---------------------+ 3 minutes later and progress still 0%. Should I just let queries gradually fetch the data from S3?

It is common for the

PROGRESS

column to stay at 0% for several minutes if your table is large or if the system is still in the initial scanning phase. Behind the scenes,

CACHE SELECT

runs like a regular query (it uses an

INSERT INTO BLACKHOLE

mechanism). Before you give up and rely on gradual fetching, you should verify if the cache is actually being populated. 1. Check if Cache is Growing Don't rely solely on the

task_runs

view. Instead, check if the actual disk usage for the cache is increasing across your nodes. Run this multiple times and see if

DISK_USED_BYTES

is rising:

sql
SELECT BE_ID, DISK_QUOTA_BYTES, DISK_USED_BYTES 
FROM information_schema.be_datacache_metrics;

If

DISK_USED_BYTES

is increasing, the warmup is working, even if the task progress hasn't updated yet. 2. Verify Real-time Write Activity You can check if the Backend (BE) nodes are currently writing data to the cache by hitting the BE's internal API (replace

BE_IP

and

BE_HTTP_PORT

with your node's info):

bash
curl http://<BE_IP>:<BE_HTTP_PORT>/api/datacache/stat

Look for

block_cache_write_bytes

and

block_cache_current_writing_count

. If these values are active/increasing, the task is healthy. 3. Why it might be "stuck" at 0% * Table Size: If the

events

table has thousands of tablets or petabytes of data, the query planning and initial metadata fetch from S3 can take time. * Resource Throttling: If the cluster is under heavy load, the warmup task (which runs at normal query priority) might be waiting for execution threads. * Progress Granularity: In some versions, the progress for

SUBMIT TASK

only updates after certain processing milestones (like a set number of tablets). Should you just let queries fetch it? Probably not, unless you have Cache Sharing enabled. * If you do NOT have Cache Sharing enabled: Gradual fetching means every single user query will be slow (cold start) until the specific data it needs is cached. * If you DO have Cache Sharing enabled: You can afford to be more relaxed. New nodes can "borrow" data from existing nodes' caches over the network, which is much faster than S3. Recommendation: Keep the

warmup_task

running for a bit longer and monitor the

DISK_USED_BYTES

. If you see no increase in disk usage after 10–15 minutes, check the BE logs (

<http://be.INFO|be.INFO>

) for any S3 connection errors or "Cache full" warnings. References • data_source/data_cache_observe.md • data_source/block_cache_warmup.md • data_source/data_cache_troubleshooting.md References • data_source/block_cache_warmup.md • data_source/data_cache_observe.md • deployment/shared_data/feature-support-shared-data.md