http://starrocks.io logo
Join Slack
Powered by
<@U0A71G31CDV> I have problems with the publishing...
# questions-and-troubleshooting
z
@Rocky I have problems with the publishing s3 tables
r
To help you better, could you clarify what you mean by "publishing S3 tables"? Are you referring to: 1. Shared-data mode (Cloud-native): Data is stored in S3, and you are seeing "Publish Version" failures or timeouts during data loading? 2. External Tables: You are creating tables that point to data already on S3 (Hive, Iceberg, or File external tables) and they aren't appearing or syncing correctly? 3. Data Loading: You are loading data from S3 into StarRocks (e.g., via Broker Load, Pipe, or 
INSERT INTO ... SELECT FROM FILES()
) and the job is stuck in the "COMMITTED" state but not "FINISHED"? If you have an error message (like a timeout or a specific "failed to publish" error), please share it. please share it. Common issues with "publishing" usually involve: * Timeouts: If the 
publish_version
phase is slow, it might be due to high ingestion concurrency. You can try increasing 
transaction_publish_version_worker_count
on the BEs. * Permissions: If using shared-data mode, the StarRocks cluster needs appropriate IAM permissions (S3:PutObject, S3:GetObject, S3:ListBucket, etc.) to write metadata and data files to the bucket. * Connectivity: Issues with the S3 endpoint or region configuration in your FE/BE config files (
aws_s3_region
, 
aws_s3_endpoint
). Looking forward to more details! References • loading/s3.md • loading/Flink-connector-starrocks.md
z
@Rocky com.starrocks.rpc.RpcException: Fail to publish version for tablets [10155, 10156, 10162, 10159]: starlet err [RequestID=0a64c61119ed1527e8314736:10001][StatusCode=404]Get object s3://dev-starrocks-fe/meta/fe919e73-9b67-4920-b15b-fad76a32c664/db10004/10152/10154/meta/0000000000000000_000000000000037D.meta error: The specified key does not exist., host: kube-starrocks-cn-1.kube-starrocks-cn-search.starrocks.svc.cluster.local at com.starrocks.lake.Utils.publishVersionBatch(Utils.java:177) ~[starrocks-fe.jar:?] at com.starrocks.transaction.PublishVersionDaemon.publishPartitionBatch(PublishVersionDaemon.java:618) ~[starrocks-fe.jar:?] at com.starrocks.transaction.PublishVersionDaemon.lambda$publishLakeTransactionBatchAsync$14(PublishVersionDaemon.java:760) ~[starrocks-fe.jar:?] at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1768) ~[?:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[?:?]
r
The error message 
starlet err [StatusCode=404] Get object ... error: The specified key does not exist
during a Publish Version task in a shared-data (cloud-native) cluster indicates that StarRocks is looking for a metadata file (
.meta
) in S3 that is missing. This is often caused by the metadata being prematurely cleaned up by the Lake Vacuum process or a synchronization issue between the FE and S3. Possible Causes and Solutions 1. Metadata Vacuumed Prematurely If the vacuum process is too aggressive, it might delete metadata files that are still needed for in-flight transactions or recent version transitions. * Fix: Increase the vacuum grace period in your FE configuration (
fe.conf
).
Copy code
properties
    lake_autovacuum_grace_period_minutes = 1440  # Increase to 24 hours (default is usually shorter)
This gives the system more time to complete long-running transactions or retries before metadata is deleted. 2. Known Bug in Older Versions There are several known issues in versions prior to 3.2.x and 3.3.x related to "file bundling" and metadata version pinning that can lead to these 404 errors during publishing or querying. * Action: Check your StarRocks version. If you are on an early 3.x version, consider upgrading to the latest maintenance release (e.g., 3.2.8+, 3.3.3+). 3. S3 Bucket Consistency or Permissions Ensure that the IAM policy for the StarRocks cluster has full 
s3:DeleteObject
, 
s3:PutObject
, and 
s3:GetObject
permissions. Occasionally, eventual consistency (though rare in modern S3) or external lifecycle policies on the bucket can cause files to disappear. * Action: Verify there are no S3 Lifecycle Rules on your bucket that might be deleting files in the 
meta/
prefix. 4. Temporary Data Corruption If the error persists for specific tablets after retries, the metadata for those tablets might be inconsistent. * Troubleshooting: 1. Check the status of the tablets:
Copy code
sql
        SHOW TABLET 10155;
2. Check if the S3 path in the error actually exists using the AWS CLI:
Copy code
bash
        aws s3 ls <s3://dev-starrocks-fe/meta/fe919e73-9b67-4920-b15b-fad76a32c664/db10004/10152/10154/meta/0000000000000000_000000000000037D.meta>
3. If the file is truly gone and publishing continues to fail, you may need to TRUNCATE or DROP and recreate the affected partition/table if you do not have a backup, as the metadata chain for that tablet is broken. Next Steps 1. Check your 
fe.log
for any 
ERROR
logs from 
LakeVacuum
around the time of the failure. 2. Try increasing the 
lake_autovacuum_grace_period_minutes
as a first step to see if it stabilizes the cluster. References • deployment/shared_data/s3.md • faq/shared_data_faq.md • administration/management/BE_configuration.md References • faq/shared_data_faq.md • deployment/shared_data/s3.md • knowledge/admin_and_operation/shared_data_garbage_cleanup.md
z
@Rocky that does not exist 0000000000000000_000000000000037D.meta what should I do?
r
If the 
.meta
file is physically missing from S3, the metadata chain for those tablets is broken. In StarRocks shared-data mode, this prevents new data from being published and often makes the table unqueryable. You can recover the table by rolling it back to its last known healthy state using the 
ADMIN REPAIR TABLE
 command. Step 1: Identify the Affected Table Use the tablet IDs from your error message (e.g., 
10155
) to find which table and partition are broken:
Copy code
sql
SHOW TABLET 10155;
Note the 
TableName
and 
PartitionName
from the output. Step 2: Attempt a Metadata Repair Run the repair command for that table. This process scans S3 for the most recent version where all metadata files still exist and rolls the table back to that point. Option A: Roll back to a consistent healthy version (Recommended) This will roll back all tablets in the partition to a single consistent point in time. You may lose the most recent successful loads, but the table will be functional again.
Copy code
sql
ADMIN REPAIR TABLE <your_table_name> 
PARTITION (<your_partition_name>)
PROPERTIES ("enforce_consistent_version" = "true");
Option B: Maximum availability (If Option A fails) If you cannot find a consistent version for the whole partition, you can allow tablets to roll back to different versions to save as much data as possible.
Copy code
sql
ADMIN REPAIR TABLE <your_table_name> 
PARTITION (<your_partition_name>)
PROPERTIES ("enforce_consistent_version" = "false");
Option C: Reset corrupt tablets (Last Resort) If the metadata is completely gone and you just want to make the table work again (accepting data loss for those specific tablets), you can allow the system to create empty tablets:
Copy code
sql
ADMIN REPAIR TABLE <your_table_name> 
PARTITION (<your_partition_name>)
PROPERTIES ("allow_empty_tablet_recovery" = "true");
Step 3: Verify the Status Check if the tablets are now healthy:
Copy code
sql
ADMIN SHOW REPLICA STATUS FROM <your_table_name> 
WHERE TabletId = 10155;
Why did this happen? To prevent this from happening again, check the following: 1. Vacuum Grace Period: The file might have been cleaned up by the background vacuum process. Increase this setting in `fe.conf`:
Copy code
properties
    lake_autovacuum_grace_period_minutes = 1440 # Default is often 1 hour; set to 24 hours.
2. Version Bug: There are known issues in versions earlier than 3.2.10 or 3.3.3 related to metadata pinning. If you are on an older version, consider upgrading to the latest maintenance release of your branch. 3. S3 Lifecycle Policies: Ensure your S3 bucket does not have lifecycle rules that automatically delete or archive files in the 
meta/
prefix. References • sql-reference/sql-statements/cluster-management/tablet_replica/ADMIN_REPAIR.md • administration/management/resource_management/Replica.md • faq/shared_data_faq.md References • faq/shared_data_faq.md • administration/management/resource_management/Replica.md
z
@Rocky how to check non healthy tablets on table in starrocks 3.5.16 shared
r
In StarRocks 3.5.16 (shared-data mode), you can check for non-healthy or problematic tablets using the following methods. Since you are in shared-data mode, "unhealthy" usually refers to metadata corruption (missing 
.meta
files in S3) or version mismatches rather than traditional replica loss. 1. Global Health Overview To see a summary of unhealthy tablets across the entire cluster:
Copy code
sql
SHOW PROC '/statistic';
Look for the 
UnhealthyTabletNum
 column. If it is greater than 0, you can drill down into a specific database:
Copy code
sql
SHOW PROC '/statistic/<db_id>';
_(You can get the 
<db_id>
from 
SHOW DATABASES
or 
SELECT SCHEMA_NAME, SCHEMA_ID FROM information_schema.schemata
)._ 2. Table-Level Health Check This is the most effective way to find tablets with version errors or status issues for a specific table:
Copy code
sql
ADMIN SHOW REPLICA STATUS FROM <table_name> 
WHERE STATUS != "OK";
In shared-data mode, even though there is technically only one "replica" per tablet, this command still works. Pay attention to: * 
Status
: Should be 
OK
. If it is 
VERSION_ERROR
or 
MISSING
, the tablet is unhealthy. * 
IsBad
: If this is 
true
, the tablet is considered corrupted. 3. Using System Tables (SQL-based) You can query the 
information_schema
to find tablets that are in an abnormal state across the cluster:
Copy code
sql
SELECT * 
FROM information_schema.be_tablets 
WHERE STATE != 'NORMAL';
4. Partition Version Mismatch (Shared-Data Specific) Since your issue involves a 404 error (missing metadata), it often manifests as a mismatch between the partition's visible version and the tablet's version. You can check partition health via:
Copy code
sql
SELECT 
    DB_NAME, TABLE_NAME, PARTITION_NAME, 
    VISIBLE_VERSION, NEXT_VERSION, MAX_CS
FROM information_schema.partitions_meta
WHERE VISIBLE_VERSION < NEXT_VERSION - 1;
_Note: A large gap between 
VISIBLE_VERSION
and 
NEXT_VERSION
or a high 
MAX_CS
(Compaction Score) can indicate that the background processes (like publishing or compaction) are stuck._ 5. Detailed Tablet Diagnosis Once you have a specific 
TabletId
from the steps above, get its full metadata path and status:
Copy code
sql
SHOW TABLET <tablet_id>;
This will provide a 
DetailCmd
. Copy and run that command (usually a 
SHOW PROC '/dbs/...'
path) to see exactly which backend is reporting the error and what the specific version mismatch is. Recommendation Given your earlier 404 metadata error, if you find tablets that are permanently "Bad" or stuck in "VERSION_ERROR", the best way to recover is the repair command:
Copy code
sql
ADMIN REPAIR TABLE <table_name> 
PROPERTIES ("enforce_consistent_version" = "true");
References • sql-reference/sql-statements/cluster-management/tablet_replica/ADMIN_SHOW_REPLICA_STATUS.md • sql-reference/information_schema/be_tablets.md • faq/operation_maintenance_faq.md • knowledge/admin_and_operation/shared_data_compaction.md References • sql-reference/information_schema/be_tablets.md • faq/operation_maintenance_faq.md • knowledge/admin_and_operation/shared_data_compaction.md
z
@Rocky [42000][1064] Fail to repair tablet metadata for 1 partition, the first 1 partition: [{partition: 10154, error: no consistent valid tablet metadata version was found, you can set enforce_consistent_version=false}]
r
The error message indicates that StarRocks cannot find a single historical version where all tablets in partition 
10154
are consistent. This happens when the metadata for different tablets has been deleted/vacuumed at different rates. To proceed, you need to use a less restrictive repair mode. Step 1: Repair with 
enforce_consistent_version = false
 This command tells StarRocks to repair each tablet individually. Each tablet will roll back to its own latest available healthy metadata file. Note: This may result in some tablets having slightly more recent data than others within the same partition, which can lead to minor data inconsistencies, but it will make the partition usable again.
Copy code
sql
ADMIN REPAIR TABLE <your_table_name> 
PARTITION (<your_partition_name>)
PROPERTIES ("enforce_consistent_version" = "false");
Step 2: If Step 1 still fails (Last Resort) If StarRocks still cannot find any valid metadata for certain tablets, you must allow the system to "reset" those tablets to an empty state to recover the rest of the partition. This will result in data loss for the specific corrupted tablets.
Copy code
sql
ADMIN REPAIR TABLE <your_table_name> 
PARTITION (<your_partition_name>)
PROPERTIES (
    "enforce_consistent_version" = "false",
    "allow_empty_tablet_recovery" = "true"
);
Step 3: Verify Recovery After running the repair command, check the status of the tablets again:
Copy code
sql
ADMIN SHOW REPLICA STATUS FROM <your_table_name> 
WHERE STATUS != "OK";
If the status is now 
OK
, try running a 
SELECT COUNT(*)
on the table to ensure it is queryable. Critical: Prevent this from happening again In StarRocks 3.3+ and 3.5.x, this 404 error is usually caused by the 
LakeVacuum
process being too aggressive. Since you have already lost metadata, you should immediately increase the vacuum grace period in your FE configuration (
fe.conf
):
Copy code
properties
# Increase to 24 hours (1440 minutes) to ensure metadata is kept 
# long enough for all nodes and background tasks to sync.
lake_autovacuum_grace_period_minutes = 1440
You should also check if your S3 Bucket Lifecycle Rules are accidentally deleting files in the 
/meta/
or 
/data/
prefixes. StarRocks must manage the lifecycle of these files itself. References • sql-reference/sql-statements/cluster-management/tablet_replica/ADMIN_REPAIR.md • faq/shared_data_faq.md • knowledge/admin_and_operation/shared_data_garbage_cleanup.md References • sql-reference/sql-statements/cluster-management/tablet_replica/ADMIN_REPAIR.md • faq/shared_data_faq.md • knowledge/admin_and_operation/shared_data_garbage_cleanup.md
z
@Rocky its internal statistics.external_column_statistics table
r
Since the table 
_statistics_.external_column_statistics
is an internal system table used for storing metadata about external tables (Hive, Iceberg, etc.), the data inside it is non-critical. If the metadata is corrupted (missing 
.meta
files) and the 
ADMIN REPAIR
command cannot find a consistent version, the most effective solution is to clear the statistics and let the system re-collect them. Follow these steps in order: 1. Try to Truncate the Table Truncating the table is the fastest way to clear the corrupt tablet information.
Copy code
sql
TRUNCATE TABLE _statistics_.external_column_statistics;
If this fails with a "tablet not found" or "meta not found" error, move to the next step. 2. Drop and Recreate (Recommended) Internal statistics tables are managed by an FE background thread. If you drop the table, StarRocks will automatically recreate the table structure (schema) within a few minutes. 1. Force-drop the corrupted table:
Copy code
sql
    DROP TABLE _statistics_.external_column_statistics FORCE;
2. Verify Recreation: Wait about 2–5 minutes and check if the table has been recreated by the FE:
Copy code
sql
    SHOW TABLES IN _statistics_;
3. Re-collect Statistics: Once the table exists again, it will be empty. You should manually trigger a collection for the external tables you use to restore performance:
Copy code
sql
    -- Example for a Hive table
    ANALYZE TABLE <catalog_name>.<db_name>.<table_name>;
3. Immediate Prevention (CRITICAL) This error occurs because the 
LakeVacuum
process deleted the metadata files from S3 while the FE/BE still thought they were needed. To prevent this from happening to your user data tables: 1. Update 
fe.conf
on all FE nodes: Increase the vacuum grace period to ensure metadata is kept longer (default is often only 1 hour).
Copy code
properties
    lake_autovacuum_grace_period_minutes = 1440
2. Restart the FE to apply the change. 3. Check S3 Lifecycle Rules: Ensure your S3 bucket does not have any "Expiration" or "Archive" rules on the folders where StarRocks stores data. StarRocks must manage the file lifecycle itself. Why this happened? In StarRocks 3.5.x shared-data mode, the system periodically cleans up old versions of metadata. If a Backend (BE) or Frontend (FE) is lagging or if a compaction job is interrupted, the vacuum process might prematurely delete a 
.meta
file that is still referenced. Increasing the grace period provides a safety buffer. References • using_starrocks/Cost_based_optimizer.md • sql-reference/sql-statements/cluster-management/tablet_replica/ADMIN_REPAIR.md • faq/shared_data_faq.md References • using_starrocks/Cost_based_optimizer.md • sql-reference/sql-statements/cluster-management/tablet_replica/ADMIN_REPAIR.md • faq/shared_data_faq.md References • sql-reference/sql-functions/json-functions/json-query-and-processing-functions/get_json_string.md • sql-reference/sql-functions/json-functions/overview-of-json-functions-and-operators.md • sql-reference/sql-functions/json-functions/json-operators.md
z
@Rocky may be I can just drop partition?
r
You can drop a partition, but for the 
_statistics_.external_column_statistics
table, it is usually better to truncate or use the specialized 
DROP STATS
 command. Here is why and how to proceed: 1. Is it a partitioned table? In StarRocks 3.x, the 
external_column_statistics
table is often not partitioned by the standard user-visible 
PARTITION
syntax (like 
p202301
). It is usually a single-partition table with multiple buckets. * The "partition 10154" in your error message refers to the internal physical partition ID, which doesn't always map to a name you can use in a 
DROP PARTITION
command. * To check if it has partitions you can drop, run:
Copy code
sql
    SHOW PARTITIONS FROM _statistics_.external_column_statistics;
If it only shows one partition (often named after the table), dropping it is equivalent to truncating the table. 2. Recommended: Use 
DROP STATS
 Instead of manually messing with the table's partitions, use the built-in command to clear the statistics. This is cleaner and ensures the FE's internal cache is also invalidated:
Copy code
sql
-- This will clear the stats for a specific external table that is causing issues
DROP STATS <catalog_name>.<db_name>.<table_name>;
3. If you still want to "Drop" the data to fix the health issue If the 
ADMIN REPAIR
failed and you just want the "unhealthy tablet" warning to go away, you have two safe options for this specific internal table: Option A: Truncate (Cleanest)
Copy code
sql
TRUNCATE TABLE _statistics_.external_column_statistics;
* Pros: Deletes all data and metadata for the table and resets the tablets. * Cons: You will lose existing statistics for all external tables (but StarRocks will automatically start re-collecting them as queries run). Option B: Drop and Force Recreation If 
TRUNCATE
also fails because of the metadata error:
Copy code
sql
DROP TABLE _statistics_.external_column_statistics FORCE;
* StarRocks FE has a background thread that monitors system tables. If it sees this table is missing, it will recreate it automatically within a few minutes. Summary Since this is an internal table for external table statistics, there is zero risk to your actual data. The only impact is that queries against external catalogs (Hive, Iceberg, etc.) might be slightly slower until the statistics are re-gathered. My recommendation: Run 
TRUNCATE TABLE _statistics_.external_column_statistics;
. It is the most direct way to resolve the 404/unhealthy tablet error for this table in the metadata. tablet issue for this specific. References • using_starrocks/Cost_based_optimizer.md • sql-reference/sql-statements/cbo_stats/DROP_STATS.md • knowledge/admin_and_operation/shared_data_compaction.md
2 Views
Open in Slack
PreviousNext
Powered by