http://starrocks.io logo
Join Slack
Powered by
<@U0A71G31CDV> im trying to update a field in a ta...
# questions-and-troubleshooting
r
@Rocky im trying to update a field in a table and consistently see 
ERROR 1064 (HY000): number of exprs is not same with slots backend [id=56006] [
. what does this mean?
r
The error 
ERROR 1064 (HY000): number of exprs is not same with slots
is a Query Plan Mismatch error. It occurs when the StarRocks Frontend (FE) generates an execution plan where the number of data values (expressions) being sent to the Backend (BE) does not match the number of target column slots defined in the plan. This is most commonly seen when performing 
UPDATE
operations on Primary Key tables. Why is this happening? 1. Plan Cache Inconsistency: StarRocks often caches query plans to improve performance. If you are running many 
UPDATE
statements with different structures or under high concurrency, the FE might accidentally reuse a cached plan that doesn't perfectly match the current statement's schema or parameters. 2. Metadata Versioning: If the table schema was recently changed or if there is high pressure on the metadata, the BE might be expecting a different number of columns than what the FE is currently providing in its serialized plan. 3. Internal Bug (Version Specific): This error was notably frequent in some older versions (e.g., 2.5.x or early 3.x) during high-frequency partial updates. How to fix it You can try the following solutions in order of complexity: 1. Disable the Plan Cache for your session This is the most common fix. By disabling the cache, you force the FE to re-generate a fresh plan for every statement, which avoids the "slot mismatch" issue.
Copy code
sql
SET enable_plan_cache = false;
After running this, try your 
UPDATE
statement again. If it works, you can apply this globally if needed (though it may slightly impact performance for high-frequency small queries):
Copy code
sql
SET GLOBAL enable_plan_cache = false;
2. Check for "Partial Updates" Configuration If you are using a Primary Key table and performing partial updates (updating only specific columns), ensure that you haven't enabled conflicting session variables or that your 
UPDATE
statement isn't inadvertently missing required primary key columns in its logic. 3. Version Upgrade If you are on an older version of StarRocks (like 2.5.x or 3.1.x), this was a known issue that has been addressed in later maintenance releases. Upgrading to the latest stable version in your major branch (e.g., 3.2.x or 3.3.x) is highly recommended. Summary: The error means the "shape" of the data being sent to the backend doesn't fit the "slots" prepared by the plan. Running 
SET enable_plan_cache = false;
is usually the fastest way to get your updates working again. References • knowledge/admin_and_operation/shared_data_garbage_cleanup.md • faq/shared_data_faq.md • faq/Others.md • quick_start/shared-data.md • deployment/shared_data/azure.md • deployment/shared_data/s3.md
r
@Rocky i get "ERROR 1193 (HY000): Getting analyzing error. Detail message: Unknown system variable 'enable_plan_cache', the most similar variables are {'enable_plan_capture', 'enable_plan_analyzer', 'enable_scan_datacache'}."
r
Apologies for the confusion—the variable name differs slightly depending on your StarRocks version. The error 
number of exprs is not same with slots
occurs when the execution plan generated by the Frontend (FE) tells the Backend (BE) to expect a certain number of columns (slots), but the actual data sent contains a different count. This "mismatch" is most common during 
UPDATE
operations on Primary Key tables. Here is how to resolve it: 1. Try the Correct Variable Name In many StarRocks versions, the plan cache variable is named 
enable_query_plan_cache
(with the word "query"). Try running:
Copy code
sql
SET enable_query_plan_cache = false;
If this works, it confirms that a stale cached plan was causing the mismatch. 2. Force the Update Mode (Primary Key Tables) If you are updating a Primary Key table, the error often stems from an automatic choice between "row mode" and "column mode" for partial updates. You can force one mode to see if it clears the conflict:
Copy code
sql
-- Try forcing Row Mode
SET partial_update_mode = 'row';

-- OR try forcing Column Mode (available in v3.1+)
SET partial_update_mode = 'column';
3. Check for Version Mismatch This error frequently appears during rolling upgrades. If some of your Backends (BEs) were upgraded but your Frontends (FEs) were not (or vice versa), the communication protocol for 
UPDATE
plans might have changed. * Run 
SHOW BACKENDS;
and 
SHOW FRONTENDS;
. * Verify that the 
Version
column is identical across all nodes. If they differ, complete the upgrade of all nodes to the same version. 4. Restart the FE (Metadata Refresh) If the above session variables do not work, it usually indicates a metadata inconsistency in the Leader FE's memory. * The quickest way to clear this state is to restart the Leader FE. This forces the metadata and query plans to be re-initialized from the image/journal. Why did my previous suggestion fail? You likely saw the "Unknown system variable" error because: 1. Missing "query" prefix: The variable is often 
enable_query_plan_cache
. 2. Version: In some versions (pre-3.x), plan caching is controlled strictly by the 
fe.conf
file (
enable_query_plan_cache = true/false
) and cannot be changed via a SQL session. Recommended next step: Try 
SET enable_query_plan_cache = false;
. If that returns the same "Unknown variable" error, please run 
SELECT version();
and I can give you the specific configuration for your version. References • faq/Deploy_faq.md • faq/Sql_faq.md • sql-reference/sql-statements/table_bucket_part_index/UPDATE.md • faq/Others.md • sql-reference/sql-statements/cluster-management/nodes_processes/SHOW_BACKENDS.md • deployment/upgrade.md • administration/management/BE_configuration.md
r
@Rocky SET partial_update_mode = 'column'; worked - why would that be?
r
Setting 
partial_update_mode = 'column'
worked because it changed the fundamental way StarRocks processes your 
UPDATE
statement, bypassing a complex "Read-on-Write" mapping that was failing. Here is the technical breakdown of why this fixed the "slots mismatch" error: 1. Row Mode (The Default/`auto` behavior) By default, StarRocks often uses Row Mode for partial updates. In this mode: * The Process: StarRocks reads the entire existing row from the disk, replaces the specific fields you want to update in memory, and then writes the complete new row back to the storage engine. * The Error: The "number of exprs is not same with slots" error occurs here because the Query Planner (Frontend) incorrectly calculated how many columns (slots) were needed for this "Read-Modify-Write" operation. If your table has 10 columns and you update 2, the Backend might be expecting a plan for all 10, but the Frontend only provided expressions for 2 (or vice-versa). 2. Column Mode (The Fix) When you forced `partial_update_mode = 'column'`: * The Process: StarRocks only writes the Primary Key and the specific columns you are updating. It does not attempt to read or rebuild the other columns in the row. * The Result: Because the execution plan is much simpler—dealing only with the updated columns—it avoids the complex internal mapping that was causing the "slot" mismatch in the first place. Why did the error happen at all? This error is usually a symptom of one of the following: * Plan Cache Inconsistency: The Frontend might have cached an execution plan for a different version of the table schema or a different 
UPDATE
structure, leading to a mismatch between the expected data "shape" and the actual data sent. * Complex Updates: If your 
UPDATE
statement involves a 
JOIN
or a subquery, the planner sometimes struggles to align the output of that join with the physical column slots of the target table in Row Mode. * Internal Bug: This specific error was a known issue in some early 3.x versions (specifically around how partial updates were handled in the pipeline engine) and was largely resolved in later maintenance releases (v3.1.4+, v3.2.x). Recommendation While forcing 
column
mode is a valid and often more performant fix (especially for updating a small number of columns in many rows), you should also consider: 1. Checking your version: If you are on an early 3.x version, upgrading to the latest patch release will likely fix the underlying planner bug. 2. Plan Cache: If you encounter this again with other queries, you can disable the plan cache for your session using:
Copy code
sql
    SET enable_query_plan_cache = false;
References • using_starrocks/SQL_plan_manager.md • administration/management/FE_configuration.md • sql-reference/sql-functions/array-functions/unnest.md • administration/management/resource_management/Replica.md • faq/Sql_faq.md • faq/Others.md • deployment/sr_operator.md
r
@Rocky im on 4.0.8. and how do i show the partial_update_mode?
r
To see the current value of the 
partial_update_mode
for your session, you can use the following command:
Copy code
sql
SHOW VARIABLES LIKE 'partial_update_mode';
Or, if you want more detail (such as whether it has been changed from its default value), you can query the system views:
Copy code
sql
SELECT * FROM information_schema.session_variables 
WHERE variable_name = 'partial_update_mode';
Why does this fix it in version 4.0.8? Even in the latest versions like 4.0.8, there is a fundamental difference in how StarRocks handles the two modes: 1. Row Mode (Default/`auto`): The system performs a "Read-on-Write." It reads the existing row, merges it with your update in memory, and writes it back. The error you saw (
number of exprs is not same with slots
) happens when the Frontend's "blueprint" for this merge doesn't perfectly match what the Backend expects. This is often triggered by specific table schemas (like those with many columns or complex types). 2. Column Mode: The system writes only the Primary Key and the specific columns you updated. It doesn't attempt to reconstruct the full row on the fly. This significantly simplifies the execution plan, which is why it bypasses the "slot mismatch" bug. Is it safe to leave it on? Yes, for version 4.0.x, keeping 
partial_update_mode = 'column'
is often better for performance if: * You are updating a small number of columns in a table with many columns. * You are performing large-scale updates. Note on 4.0.8: You are on a very cutting-edge version! If this issue persists without the manual override, it might be worth reporting to the StarRocks community as a potential regression in the 
auto
planner logic for the 4.0 branch. References • sql-reference/sql-statements/cluster-management/config_vars/SHOW_VARIABLES.md • sql-reference/System_variable.md • sql-reference/information_schema/session_variables.md • sql-reference/sql-statements/table_bucket_part_index/UPDATE.md • sql-reference/information_schema/verbose_session_variables.md • release_notes/release-4.0.md • administration/management/FE_configuration.md • using_starrocks/SQL_plan_manager.md • administration/management/BE_configuration.md • faq/Sql_faq.md • faq/Others.md • administration/management/resource_management/Replica.md • deployment/sr_operator.md • sql-reference/sql-functions/array-functions/unnest.md • release_notes/release-2.0.md References • release_notes/release-4.0.md • release_notes/release-2.0.md
r
@Rocky another error:
Copy code
2026-04-16 14:58:53.329Z ERROR (starrocks-mysql-nio I/O-2|142) [MySQLReadListener.handleEvent():97] Unexpected error in MySQLReadListener
java.net.SocketException: Connection reset
	at sun.nio.ch.SocketChannelImpl.throwConnectionReset(SocketChannelImpl.java:394) ~[?:?]
	at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:426) ~[?:?]
	at org.xnio.nio.NioSocketConduit.read(NioSocketConduit.java:289) ~[xnio-nio-3.8.16.Final.jar:3.8.16.Final]
	at org.xnio.conduits.ConduitStreamSourceChannel.read(ConduitStreamSourceChannel.java:127) ~[xnio-api-3.8.16.Final.jar:3.8.16.Final]
	at com.starrocks.mysql.nio.MySQLReadListener.handleEvent(MySQLReadListener.java:58) ~[starrocks-fe.jar:?]
	at com.starrocks.mysql.nio.MySQLReadListener.handleEvent(MySQLReadListener.java:36) ~[starrocks-fe.jar:?]
	at org.xnio.ChannelListeners.invokeChannelListener(ChannelListeners.java:92) ~[xnio-api-3.8.16.Final.jar:3.8.16.Final]
	at org.xnio.conduits.ReadReadyHandler$ChannelListenerHandler.readReady(ReadReadyHandler.java:66) ~[xnio-api-3.8.16.Final.jar:3.8.16.Final]
	at org.xnio.nio.NioSocketConduit.handleReady(NioSocketConduit.java:89) ~[xnio-nio-3.8.16.Final.jar:3.8.16.Final]
	at org.xnio.nio.WorkerThread.run(WorkerThread.java:603) ~[xnio-nio-3.8.16.Final.jar:3.8.16.Final]
r
The error 
java.net.SocketException: Connection reset
appearing in your 
fe.log
under 
MySQLReadListener
indicates that a connection between a client and the StarRocks Frontend (FE) was abruptly closed by the client-side or an intermediate network device (like a load balancer). In this scenario, StarRocks is acting as the MySQL server. It was expecting to read more data from the socket, but the "peer" (the client) sent a TCP RST packet to terminate the connection. Common Causes for v4.0.8 1. Client-Side Timeouts: * If you are running a long query, the client (e.g., a JDBC driver, Python script, or MySQL Workbench) might have a shorter timeout than the time StarRocks takes to process the query. When the client's timeout hits, it closes the connection while the FE is still trying to talk to it. * Check: Does this error coincide with specific long-running queries? 2. Load Balancer / Proxy Disconnects: * If you have an HAProxy, Nginx, or a Cloud Load Balancer (AWS ELB, etc.) between the client and StarRocks, the proxy's 
idle_timeout
might be reached. Most proxies default to 60 seconds. * Check: Increase the idle timeout on your proxy/load balancer. 3. FE JVM "Stop-the-World" GC Pauses: * If the FE is under high memory pressure and triggers a long Full GC pause, it stops responding to heartbeats or network requests. The client may then time out and reset the connection. * Check: Look for 
Full GC
or 
pause
messages in 
fe.gc.log
around the same timestamp (
14:58:53
). 4. Network Instability: * Physical network issues or firewall rules that kill "idle" TCP connections. * Action: You can try increasing the TCP keep-alive settings on the OS or adjusting StarRocks variables:
Copy code
sql
        -- Increase idle timeouts (default is usually 28800s / 8 hours)
        SET GLOBAL wait_timeout = 28800;
        SET GLOBAL interactive_timeout = 28800;
Troubleshooting Steps 1. Check OS Metrics: Run these commands on the FE host to see if the OS is dropping connections at the backlog level:
Copy code
bash
    netstat -s | grep -i LISTEN
    netstat -s | grep TCPBacklogDrop
If 
TCPBacklogDrop
is increasing, you may need to increase 
net.core.somaxconn
in sysctl. 2. Correlate with Query Logs: Check 
fe.audit.log
for queries that started just before 
14:58:53
and see if they were interrupted or had very long durations. 3. Client Logs: Check the logs of the application or tool that was connected to StarRocks at that time. It will likely show a "Socket closed" or "Communication link failure" error. Summary: This error is rarely a "bug" in StarRocks itself; it is a notification that the external connection was killed while StarRocks was still using it. Focus your investigation on the client and the network path (Load Balancers/Firewalls). References • faq/operation_maintenance_faq.md • loading/Json_loading.md • faq/loading/synchronize_mysql_into_sr.md
r
@Rocky i wound up setting enable_collect_full_statistic = false; its been 16 minutes since the last FULL job ran and no samples have kicked off. it is possible it messed up my scheduled analyze?
r
Disabling 
enable_collect_full_statistic
can definitely impact your scheduled 
ANALYZE
tasks because of how StarRocks decides between FULL and SAMPLE collection. Here is why your scheduled jobs might have gone quiet: 1. The "Skip" vs. "Fallback" Logic By default, StarRocks uses an internal heuristic to decide which type of collection to run: * Small Tables (or low data volume change): The system triggers a FULL collection. * Large Tables (or high volume change): The system triggers a SAMPLE collection. When you set 
enable_collect_full_statistic = false
, you are specifically telling the system not to run automatic FULL collections. If your tables are currently in a state where the "Automatic Collection" logic determines they need a FULL update (based on health/size), the system will simply skip the task rather than automatically falling back to a SAMPLE job. 2. The Health Thresholds Automatic collection is triggered by "health" scores: * Health < 0.8 (
statistic_auto_collect_ratio
): Triggers a collection job. * Health < 0.3 (
statistic_auto_collect_sample_threshold
): Triggers a SAMPLE job. * Health between 0.3 and 0.8: Usually triggers a FULL job. If your tables have a health score between 0.3 and 0.8, the scheduler wants to run a FULL job. Since you've disabled FULL jobs, nothing happens. It won't trigger a SAMPLE job until the health drops below 0.3. 3. Timing and Intervals * Check Interval: The default interval for checking if a table needs statistics is 600 seconds (10 minutes) (controlled by 
statistic_collect_interval_sec
). * Large Table Interval: If a table is considered "large," the interval between automatic collections can be as long as 12 hours (
statistic_auto_collect_large_table_interval
). Since it has only been 16 minutes, you may have only missed one "check" cycle. What should you do? If you want to avoid the overhead of FULL collections but still want automatic statistics, instead of disabling the feature entirely, you should adjust the threshold for when StarRocks switches from FULL to SAMPLE: 1. Re-enable the setting:
Copy code
sql
    ADMIN SET FRONTEND CONFIG ("enable_collect_full_statistic" = "true");
2. Lower the "Full Collect" size limit: Lowering this threshold forces the system to use SAMPLE collection for much smaller tables.
Copy code
sql
    -- Default is 100GB. Set it to a smaller value (e.g., 5GB) to force Sample more often.
    ADMIN SET FRONTEND CONFIG ("statistic_max_full_collect_data_size" = "5368709120");
3. Check the status: To see what the scheduler is thinking, run:
Copy code
sql
    SHOW ANALYZE STATUS;
Look for the 
REASON
column; it will often tell you if a job was skipped or why it hasn't started. Summary: You haven't "permanently messed up" the schedule, but you have blocked the path the scheduler was likely trying to take. If you want automatic updates to resume for those tables, you either need to re-enable FULL collection or wait for the data to change enough that the health drops below the SAMPLE threshold (0.3). References • using_starrocks/Cost_based_optimizer.md
r
@Rocky that doesnt make sense - im set for every 10 minutes, shouldnt it just kick off a sample job instead of full due to my disabling enable_collect_full_statistic? its kicking off NO jobs
r
In StarRocks v4.0.8, setting 
enable_collect_full_statistic = false
does not stop the scheduler; it is designed to change the behavior of the built-in automatic collection tasks from FULL to SAMPLE. However, if you are seeing "no jobs kicking off" for 16 minutes, it is likely due to the interaction between the Health Thresholds and the Collection Intervals rather than the setting being broken. 1. The "Health" Threshold (Why nothing is running yet) The automatic scheduler only kicks off a job when a table's "Health" drops below a certain level. * Trigger Threshold: By default, a table must have health < 0.8 (
statistic_auto_collect_ratio
) to trigger any collection. * The "Last Run" factor: You mentioned a FULL job ran 16 minutes ago. A FULL job resets the table health to 1.0 (100%). * The Wait: Unless you have ingested a massive amount of data (roughly 20%+ of the total rows/partition) in the last 16 minutes, the system still considers the statistics "Healthy" (> 0.8) and will deliberately skip the collection during its 10-minute check. 2. The "Large Table" Throttle Even if the health is low, StarRocks throttles how often it collects statistics for large tables to prevent resource exhaustion: * Small Tables (< 5GB): Collected as soon as health drops. * Large Tables (> 5GB): Governed by 
statistic_auto_collect_large_table_interval
. The default is 43,200 seconds (12 hours). * Impact: If your table is larger than 5GB (default 
statistic_auto_collect_small_table_size
), the system will only run one automatic collection every 12 hours, regardless of how often it checks for updates. 3. Interaction with "Predicate Columns" Starting in v3.x and optimized in v4.0, StarRocks uses Predicate Column logic. * If 
enable_collect_full_statistic
is 
false
, the system switches to Sampled mode. * However, if the table has "Predicate Columns" (columns used in 
WHERE
, 
JOIN
, 
GROUP BY
), StarRocks may still try to perform a Full collection on ONLY those specific columns to ensure join performance, while sampling the rest. This is a hybrid mode that is more efficient than a full table scan but might look like it's "thinking" longer before starting. How to verify what is happening To see why the scheduler is ignoring your tables, run this command:
Copy code
sql
SHOW ANALYZE STATUS;
Look for your table and check the following columns: 1. 
HEALTH
: If this is 
1.0
or 
0.9
, no job will start. 2. 
REASON
: This column will literally tell you why a job was skipped (e.g., "Health is healthy" or "Large table interval not reached"). 3. 
PROPERTIES
: Ensure the task type has correctly switched to 
SAMPLE
. Recommendation If you want to force the system to be more aggressive with Sampled jobs: 1. Lower the Large Table Interval (if your tables are > 5GB):
Copy code
sql
    ADMIN SET FRONTEND CONFIG ("statistic_auto_collect_large_table_interval" = "3600"); -- 1 hour
2. Verify the Health Calculation: Ensure 
statistic_partition_healthy_v2
is 
true
(default in 4.0), which uses a more modern row-count change ratio to trigger updates. Summary: Your scheduled analyze isn't "messed up," it's just idle because the last FULL job made the table "too healthy" to require an update so soon. Check 
SHOW ANALYZE STATUS
to confirm the next scheduled check. References • using_starrocks/Cost_based_optimizer.md • administration/management/FE_configuration.md • sql-reference/sql-statements/data-manipulation/ANALYZE_TABLE.md • sql-reference/sql-statements/cluster-management/config_vars/SHOW_VARIABLES.md
r
@Rocky before this change it was consistently full analyzing every 10 minutes
r
According to StarRocks' internal logic, disabling FULL statistics collection can indeed stop the 10-minute cycle you were seeing. Here is exactly why that happened and how the scheduler is currently "thinking." 1. The "Gap" in Health Thresholds The automatic scheduler uses two different health thresholds to decide whether to run a job and what type of job to run: * Trigger Threshold (
statistic_auto_collect_ratio
): Default is 0.8. If health drops below this, the system wants to start an analyze job. * Sample Threshold (
statistic_auto_collect_sample_threshold
): Default is 0.3. Here is the logic flow: 1. Health drops to 0.7 (below the 0.8 trigger). 2. The scheduler asks: "Is health < 0.3?" -> No. 3. Conclusion: "I should run a FULL collection." 4. The Conflict: You have 
enable_collect_full_statistic = false
. 5. Result: The system determines it needs a FULL job but is forbidden from running it. Instead of automatically "falling back" to a SAMPLE job at the 0.8 level, it simply skips the task. It will only trigger a SAMPLE job automatically once the health drops below 0.3. 2. Why it was running every 10 minutes before Before you changed the setting, the system saw the health was below 0.8 every 10 minutes. Since FULL was allowed, it would run the job, reset the health to 1.0, and then 10 minutes later (after more data ingestion), the health would drop back below 0.8 and repeat. By disabling FULL, you've essentially created a "dead zone" between 0.3 and 0.8 health where no automatic jobs will run. 3. The "Large Table" Interval If your table is larger than 5GB (the default 
statistic_auto_collect_small_table_size
), another rule kicks in: * Small Tables (< 5GB): Can be collected as often as the 10-minute check allows (
statistic_auto_collect_small_table_interval
is 0 by default). * Large Tables (> 5GB): Are throttled to run only once every 12 hours (
statistic_auto_collect_large_table_interval
). If your table is > 5GB, it's possible that even if health drops below 0.3, it will still wait for that 12-hour window. How to fix this and get SAMPLE jobs running every 10 mins If you want the frequency of the 10-minute cycle but only using SAMPLE collection, you should adjust the thresholds so that SAMPLE kicks in earlier: 1. Lower the "Sample Threshold" so it matches the "Trigger Threshold":
Copy code
sql
    -- This tells SR to use SAMPLE collection as soon as health drops below 0.8
    ADMIN SET FRONTEND CONFIG ("statistic_auto_collect_sample_threshold" = "0.8");
2. Adjust Large Table Interval (if your table is > 5GB):
Copy code
sql
    -- Allow large tables to be analyzed every 10 minutes (600s) instead of 12 hours
    ADMIN SET FRONTEND CONFIG ("statistic_auto_collect_large_table_interval" = "600");
3. Re-enable Full Statistic (Optional but recommended with the above changes): If you set the sample threshold to 0.8, the system will choose SAMPLE for almost everything anyway, but keeping the main switch 
true
ensures the scheduler doesn't "skip" the logic entirely. To verify what's happening right now: Run 
SHOW ANALYZE STATUS;
. If you see your table listed with a status like 
PENDING
or if it's missing, it's because the health hasn't reached the 0.3 "Sample" trigger yet. References • administration/management/FE_configuration.md • using_starrocks/Cost_based_optimizer.md
r
@Rocky with 
enable_collect_full_statistic
on would i expect it to ever run a full scan on a table automatically?
r
Yes, with 
enable_collect_full_statistic = true
 (the default), StarRocks will automatically run a full scan on your tables under specific conditions. In fact, for most tables, a full scan is the default behavior. Here is exactly when the system will choose a Full Scan over a Sampled Scan: 1. When Health is between 0.3 and 0.8 The scheduler uses "Health" (1.0 = perfect, 0.0 = completely stale) to decide what to do: * Health < 0.8: The system triggers an automatic collection. * Health is between 0.3 and 0.8: The system chooses Full Collection. * Health drops below 0.3: The system switches to Sampled Collection (to save resources since the data has changed so much). 2. When the Table is "Small" * Size Threshold: Tables smaller than 100 GB (controlled by 
statistic_max_full_collect_data_size
) will default to a Full Scan. * Small Table Speed: If a table is under 5 GB (
statistic_auto_collect_small_table_size
), it can be analyzed as often as every 10 minutes. * Large Table Throttle: If a table is over 5 GB but under 100 GB, it will still get a Full Scan, but only once every 12 hours (
statistic_auto_collect_large_table_interval
). 3. When you use "Predicate Columns" If your table is very wide (many columns) but you only query a few of them, StarRocks might perform a Full Collection on just those "Predicate Columns" (the ones used in WHERE/JOIN/GROUP BY) while skipping or sampling the others. This is still a "full scan" for those specific columns. ────────── Why your 10-minute cycle disappeared When you set 
enable_collect_full_statistic = false
, you disabled the Default Auto Collection Task. Even though there are settings for "Sampled" collection, they are actually part of the same logic engine. By turning that switch off, you effectively told the scheduler: "Do not run the built-in background job that manages automatic full/sample transitions." If you want the system to run automatic scans but NEVER do a full table scan, do this instead: 1. Keep 
enable_collect_full_statistic = true
. 2. Set the sample threshold to be the same as the trigger threshold:
Copy code
sql
    -- Trigger any collection at 0.8 health, and make it a SAMPLE job immediately
    ADMIN SET FRONTEND CONFIG ("statistic_auto_collect_sample_threshold" = "0.8");
3. Lower the maximum size for full scans to something tiny (e.g., 100 MB):
Copy code
sql
    ADMIN SET FRONTEND CONFIG ("statistic_max_full_collect_data_size" = "104857600");
This forces the "Default Auto Task" to remain active, but ensures it almost always chooses the Sample method for your tables. References • using_starrocks/Cost_based_optimizer.md • administration/management/FE_configuration.md
r
@Rocky another one to debug: 2026-04-16 175942.981Z ERROR (publish-task-11631|331180) [PublishVersionDaemon.publishPartitionBatch():690] Fail to publish partition 34254 of txnIds [423694, 423724, 423725, 423695, 423756, 423749, 423748]: com.starrocks.rpc.RpcException: Fail to publish version for tablets:[], error msg: link rpc channel failed, host: starrocks-shared-data-cluster-cn-2.starrocks-shared-data-cluster-cn-search.starrocks.svc.cluster.local at com.starrocks.lake.Utils.sendAggregatePublishVersionRequest(Utils.java:347) ~[starrocks-fe.jar:?] at com.starrocks.lake.Utils.aggregatePublishVersion(Utils.java:362) ~[starrocks-fe.jar:?] at com.starrocks.transaction.PublishVersionDaemon.publishPartitionBatch(PublishVersionDaemon.java:676) ~[starrocks-fe.jar:?] at com.starrocks.transaction.PublishVersionDaemon.lambda$publishLakeTransactionBatchAsync$15(PublishVersionDaemon.java:818) ~[starrocks-fe.jar:?] at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1768) ~[?:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[?:?] at java.lang.Thread.run(Thread.java:840) ~[?:?]
r
The error 
com.starrocks.rpc.RpcException: Fail to publish version for tablets:[], error msg: link rpc channel failed
indicates that the FE (Frontend) was unable to establish a network connection to the specific Compute Node (CN) mentioned in the logs (
starrocks-shared-data-cluster-cn-2...
) during the transaction publishing phase. This is a critical failure because it prevents data from becoming visible after it has been loaded. Here is how to debug and resolve this: 1. Identify the Target Node State The host in your log is a Kubernetes Headless Service FQDN for the 3rd CN pod (index 2). * Check Pod Status: Run 
kubectl get pods -n <your-namespace>
and verify if 
starrocks-shared-data-cluster-cn-2
is in a 
Running
and 
Ready
state. * Check for Restarts: If the 
RESTARTS
count is high, the CN may be OOMing (Out of Memory) or crashing, causing the RPC channel to "link fail" because the process is gone. 2. Verify Network Connectivity from FE The FE must be able to reach the CN's RPC port (usually 8060 for 
brpc_port
in shared-data mode). * Test Connection: Exec into the FE pod and try to reach the CN:
Copy code
bash
    kubectl exec -it <fe-pod-name> -- nc -vz starrocks-shared-data-cluster-cn-2.starrocks-shared-data-cluster-cn-search.starrocks.svc.cluster.local 8060
* If it fails: You have a Kubernetes networking issue (DNS resolution failure, NetworkPolicy blocking traffic, or the CN service/pod is misconfigured). 3. Analyze CN Logs Check the logs of the problematic CN to see if it was overwhelmed or rejected the connection:
Copy code
bash
kubectl logs starrocks-shared-data-cluster-cn-2 -c cn
Look for "Too many open files," OOM killer messages, or "brpc" errors. 4. Why the error is a "Batch" failure Your log mentions 
publishPartitionBatch
and a list of 
txnIds
. In StarRocks 3.2+, the 
lake_enable_batch_publish_version
setting (enabled by default) groups multiple transaction commits together to improve performance. * If the connection to the CN fails during this batch operation, all transactions in that batch (the 7 txnIds in your log) will fail to publish and remain in the 
COMMITTED
state (not visible). * The system will automatically retry, but if the network/CN issue persists, your loading will appear "stuck" at 99% or completed but with no data appearing in the table. 5. Potential Quick Fixes * If the CN is healthy but the error persists: It might be RPC congestion. You can try reducing the batch size in the FE configuration:
Copy code
sql
    ADMIN SET FRONTEND CONFIG ("lake_batch_publish_max_version_num" = "5"); -- Default is 10
* Restart the CN: Sometimes a specific CN's RPC thread pool can hang. Restarting the pod 
starrocks-shared-data-cluster-cn-2
often clears the "link rpc channel failed" state. Summary: This isn't a syntax or logic error; it's a network/process reachability failure. The FE is trying to finalize a data load, but the CN responsible for that partition's metadata is unreachable at the network level. References • administration/management/FE_configuration.md • faq/shared_data_faq.md • knowledge/usage/load/shared_data_batch_ingestion.md • faq/loading/Loading_faq.md References • faq/Deploy_faq.md • administration/management/FE_configuration.md • administration/Meta_recovery.md • faq/shared_data_faq.md
r
@Rocky is this related? Not found: starlet err [RequestID=GN4PQ21MSM5WBYMZ][StatusCode=404]Get object s3:.... error: The specified key does not exist. be/src/storage/protobuf_file.cpp:175 value_or_err_L175 be/src/storage/lake/tablet_manager.cpp:412 corrupted_tablet_meta_handler(s, metadata_location) be/src/storage/lake/vacuum.cpp:483 collect_alive_bundle_files(tablet_mgr, tablet_infos, max_vacuum_version, root_dir, &bundle_file_deleter) be/src/storage/lake/vacuum.cpp:606 vacuum_tablet_metadata(tablet_mgr, root_loc, tablet_infos, min_retain_version, grace_timestamp, request.enable_file_bundling(), &vacuumed_files, &vacuumed_file_size, &vacuumed_version, &extra_file_size, retain_versions)
r
Yes, these two errors are directly related. They are both symptoms of a communication and consistency failure involving Partition 34254 in your shared-data (lake) cluster. Here is how they are connected and what the sequence of events likely was: 1. The Relationship: Partition 34254 * Error A (RPC Failure): The FE tried to tell the Compute Node (CN) to finalize ("publish") data for partition 
34254
. Because the RPC channel failed, the FE doesn't know for certain if the metadata was successfully updated on S3 or not. * Error B (Vacuum 404): The Vacuum process (the background cleaner) is looking at the same partition 
34254
. It is trying to read a metadata file (
..._00000000000085CF_0000000000006AB1.meta
) that the system thinks should exist, but it's physically missing from S3. 2. The Root Cause: Inconsistent State In StarRocks shared-data mode, the "Publish" phase writes new metadata files to S3 to make your loaded data visible. * Scenario: If the RPC connection dropped while the metadata was being managed, or if a previous attempt failed and was partially cleaned up, the FE's internal state might still point to a metadata version that was never fully committed or was accidentally deleted by a concurrent vacuum job. * The "Race" Condition: If your 
lake_autovacuum_grace_period_minutes
is set to a very low value, the vacuum process might have deleted the "old" metadata before the "new" metadata (from your failed RPC publish) was successfully finalized. 3. Why this is happening together The 
link rpc channel failed
suggests your CN node (specifically 
cn-2
) is either: 1. Crashing/Restarting: (Check 
kubectl get pods
for restarts). 2. Under Heavy Load: High CPU/Memory pressure is causing the RPC threads to hang, leading to both failed publishes and corrupted vacuum cycles. How to Fix This Step 1: Increase the Vacuum Grace Period If you are seeing 404s on 
.meta
files, it often means the vacuum is too aggressive. Increase the time StarRocks waits before deleting old metadata files:
Copy code
sql
-- Increase to 2 hours (120 minutes) to give the system more breathing room
ADMIN SET FRONTEND CONFIG ("lake_autovacuum_grace_period_minutes" = "120");
Step 2: Check CN Health The "link rpc channel failed" is a network/process level error. * Check the logs of 
starrocks-shared-data-cluster-cn-2
for OOM (Out of Memory) kills. * Ensure the CN has enough memory. In shared-data mode, the CN manages the metadata cache; if it runs out of memory, it will drop RPC connections. Step 3: Clear the "Stuck" Transactions The transactions listed in your first error (
423694, 423724
, etc.) may be in a 
PREPARED
or 
COMMITTED
state but not 
VISIBLE
. * Check their status: 
SHOW BACKEND ACTIONS;
(look for long-running publish tasks). * Usually, the FE will retry these automatically. If they remain stuck for hours, the partition metadata may be corrupted, and you may need to truncate the affected partition or restore from a backup. Summary: The RPC failure is the primary issue (network/resource instability), and the Vacuum 404 is the secondary issue (the system trying to clean up a partition that is in an inconsistent state because of that failure). Fix the stability of 
cn-2
first, and the 404s will likely stop occurring for new data. References • faq/shared_data_faq.md • integrations/batch/SparkLoad.mdx • administration/Meta_recovery.md • integrations/batch/Spark-connector-starrocks.mdx • faq/loading/Routine_load_faq.md • faq/Others.md • faq/datalake.mdx
r
@Rockymysql> SHOW BACKEND ACTIONS; ERROR 1064 (HY000): Getting syntax error at line 1, column 13. Detail message: Unexpected input 'ACTIONS', the most similar input is {'BLACKLIST'}.
r
That error occurred because 
SHOW BACKEND ACTIONS
is not a valid StarRocks command. The system likely suggested 
BLACKLIST
because it’s the only similar keyword it recognizes for that specific syntax branch. To debug the state of your backends and those specific transactions, use these commands instead: 1. Check Backend Health To see if 
cn-2
is currently alive or reporting errors to the FE:
Copy code
sql
SHOW BACKENDS;
-- OR for a detailed view
SHOW PROC '/backends'\G
Look at the 
Alive
column and the 
ErrMsg
column to see if the FE is struggling to communicate with that specific CN. 2. Check the Status of the "Stuck" Transactions You have a list of transaction IDs from your log (
423694, 423724, ...
). You can check their exact state (e.g., if they are 
COMMITTED
but not 
VISIBLE
):
Copy code
sql
-- Replace 423694 with one of the IDs from your error log
SHOW TRANSACTION WHERE id = 423694;
* COMMITTED: The data is written, but the "Publish" phase failed (consistent with your RPC error). * VISIBLE: The transaction eventually succeeded. * ABORTED: The transaction failed and was rolled back. 3. Check Partition Consistency (Shared-Data Only) Since you are using shared-data mode and saw a 
404 Not Found
for a 
.meta
file, check the version status of the affected partition:
Copy code
sql
-- Find the database name and table for partition 34254
SELECT * FROM information_schema.partitions_meta WHERE partition_id = 34254;
_Compare 
VISIBLE_VERSION
and 
NEXT_VERSION
. if there is a large gap and transactions are stuck in 
COMMITTED
, the FE is failing to move the version forward._ Summary of what happened: The "Publish Version" is a background task handled by the FE. It sends an RPC to the CN to tell it to finalize the metadata. 1. RPC Failure: The FE tried to talk to 
cn-2
, but the "channel failed" (network glitch or CN crash). 2. Transaction Hangs: The transactions stayed in 
COMMITTED
state. 3. Vacuum Race: Because the transactions didn't finish, the Vacuum process might have looked at the partition, seen "old" files it thought were no longer needed, and deleted them—or it's looking for a metadata file that failed to upload correctly during the RPC drop, resulting in your S3 404 error. Recommendation: Restart the CN pod 
starrocks-shared-data-cluster-cn-2
. This usually clears the RPC "link failed" state and allows the FE to successfully retry the publish for those stuck transactions. References • knowledge/admin_and_operation/shared_data_compaction.md
r
@Rocky can you now give context on this error? it appeared when we tried to upgrade from 4.0.8 to 4.1.1:
Copy code
Fail to publish partition 1043403 of txnIds [1457644, 1457690, 1458529, 1458530, 1458528, 1458527, 1458660, 1459497, 1459498]:
com.starrocks.rpc.RpcException: Fail to publish version for tablets:[[1043406, 1043407, 1043405, 1043404, 1043411, 1043409, 1043408, 1043410]], error msg: Inconsistent bundle_file_offsets across txn logs for tablet 1043410: some logs have offsets, some don't. Cannot safely merge rowsets.,
r
This error is a metadata compatibility conflict triggered by the significant structural changes introduced in StarRocks 4.1 for shared-data (lakehouse) clusters. What the Error Means The error 
Inconsistent bundle_file_offsets across txn logs
means that the BE (Compute Node) is trying to "publish" a batch of transactions for a specific tablet, but the metadata within those transactions is mismatched: * "Some logs have offsets, some don't": Some transactions in the current publish queue were written using an older metadata format (likely by your previous 4.0.8 version), while others are using the new 4.1.x format. * "Cannot safely merge rowsets": In StarRocks 4.1, a major feature called "Large-Capacity Tablet Support" was introduced. This feature enables intra-tablet parallelism and changes how segment files are tracked via "bundle file offsets." The system has detected that it cannot merge these two different metadata types into a single consistent version without risking data corruption, so it aborts the publish task. Context: The 4.1 Upgrade StarRocks 4.1 is a major leap for shared-data mode. It introduces: 1. High-Capacity Tablets (100GB target): To support this, the metadata layer was rewritten to allow multiple threads to work on a single tablet simultaneously. 2. Automatic Tablet Splitting/Merging: This requires a more flexible metadata structure (offsets) within the transaction logs. 3. File Bundling: An optimization to group small metadata/data files in S3. The error occurs because you have "inflight" transactions that were started or committed in version 4.0.8 but are being "Published" (finalized) by version 4.1.1. How to Resolve This 1. Allow the System to "Drain" Old Transactions The best approach is to stop all new load jobs and allow the system to attempt to finish publishing existing transactions. If the error persists, it means the "batch" of transactions currently being published is a mixture of old and new. * Check for stuck transactions: 
SHOW TRANSACTION WHERE id IN (1457644, 1457690, ...);
* If they remain in a 
COMMITTED
state and keep failing, you may need to abort the older transactions and re-run the loads using the 4.1.1 logic. 2. Temporary Workaround: Disable File Bundling You can try to force the system to use the simpler metadata format by disabling the bundling feature temporarily. Execute this on the FE:
Copy code
sql
ADMIN SET FRONTEND CONFIG ("lake_enable_file_bundling" = "false");
Note: This might not resolve the error if the logs were already written with the inconsistent offsets, but it can prevent new ones from being generated in that format during the transition. 3. Verify Upgrade Path The 4.1 release notes explicitly warn that downgrading from 4.1 to versions below 4.0.6 is not supported due to these exact metadata changes. Ensure that: * All FE and BE/CN nodes are successfully running 4.1.1. A "mixed cluster" (some 4.0 nodes, some 4.1 nodes) will frequently trigger this error. * If you are in a Kubernetes environment, ensure the 
image
has been updated for all components and pods have restarted. Recommendation If the error does not clear after a few retries or after stopping new loads, the metadata for Tablet 1043410 (in Partition 1043403) is in an inconsistent state. You may need to: 1. Identify the table: 
SELECT * FROM information_schema.partitions_meta WHERE partition_id = 1043403;
2. Truncate and Reload that specific partition if it's a small amount of data, as this will clear the corrupted transaction logs and start fresh with the 4.1.1 metadata format. Summary: This is a "version gap" issue. The new 4.1.1 engine is seeing "legacy" 4.0.8 transaction logs and refusing to merge them because the underlying storage format for tablets changed to support much larger data larger volumes. References • administration/Meta_recovery.md
2 Views
Open in Slack
PreviousNext
Powered by