http://starrocks.io logo
Join Slack
Powered by
<@U0A71G31CDV> shared-data cluster running 4.1.3 u...
# questions-and-troubleshooting
a
@Rocky shared-data cluster running 4.1.3 under kubernetes-operator 1.11.7 has been upgraded in place to run 4.1.6. How do I clear the stale 2026-10-21 193730.062-04:00 ERROR (publish-task-3427|8826) [PublishVersionDaemon.publishPartitionBatch():767] Fail to publish partition 92632 of txnIds [8365]: com.starrocks.rpc.RpcException: Fail to publish version for tablets [92635, 92634]: starlet err [RequestID=][StatusCode=404]Get object s3://starrocks/d58a2ae0-b495-4997-b104-7f3a0b9ae0d6/db10353/92618/92632/log/00000000000169DA_00000000000020AD.log error: The specified key does not exist., host: kube-sr-cn-0.kube-sr-cn-search.sr.svc.mycorp.com at com.starrocks.lake.Utils.publishVersionBatch(Utils.java:193) ~[fe-core-4.1.jar:?] at com.starrocks.lake.Utils.publishVersionBatch(Utils.java:128) ~[fe-core-4.1.jar:?] at com.starrocks.transaction.PublishVersionDaemon.publishPartitionBatch(PublishVersionDaemon.java:738) ~[fe-core-4.1.jar:?] at com.starrocks.transaction.PublishVersionDaemon.lambda$publishLakeTransactionBatchAsync$17(PublishVersionDaemon.java:1102) ~[fe-core-4.1.jar:?] at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1768) ~[?:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[?:?] at java.lang.Thread.run(Thread.java:840) ~[?:?] left from previous version runs?
r
The error message 
StatusCode=404 Get object ... error: The specified key does not exist
indicates that your shared-data cluster is attempting to publish a transaction (TxnID 8365), but the required metadata or redo log file is missing from your S3 bucket. This can happen if a transaction was interrupted during the upgrade or if a backend cleanup was inconsistent. Since you are running version 4.1.6, you can use the Cloud-Native Repair feature introduced in StarRocks 4.1 specifically to handle lost files in shared-data mode. 1. Identify the Table and Partition First, identify which table and partition name correspond to the IDs in your error log (Partition ID 
92632
):
Copy code
sql
SELECT db_name, table_name, partition_name 
FROM information_schema.partitions_meta 
WHERE partition_id = 92632;
2. Run the Repair Command Use the 
ADMIN REPAIR TABLE
command. This will prompt the FE to scan historical metadata in S3 and roll back the partition to the latest version where all files are actually present. Strict Consistency (Recommended): This rolls back all tablets in the partition to the most recent uniform version available in S3.
Copy code
sql
ADMIN REPAIR TABLE <table_name> PARTITION (<partition_name>);
Maximum Availability (If data loss is preferred over downtime): If you want to salvage as much as possible even if tablets end up on different versions:
Copy code
sql
ADMIN REPAIR TABLE <table_name> PARTITION (<partition_name>)
PROPERTIES (
    "enforce_consistent_version" = "false"
);
3. Verify the Fix After running the repair, monitor the partition status. The 
VisibleVersion
should stabilize, and the errors in the FE log for that partition should stop.
Copy code
sql
SHOW PARTITIONS FROM <table_name> WHERE PartitionName = '<partition_name>';
Why did this happen? In shared-data mode, if a 
PublishVersion
task fails because a file is missing, the transaction remains in a "publishing" state and blocks subsequent version increments for that partition. The upgrade likely left an orphaned transaction state in the FE's image while the corresponding temporary files in S3 were cleaned up or never fully uploaded. The 
ADMIN REPAIR
command is the official way to "force clear" these stale, broken states by resetting the metadata to a known-good point in S3. For more details on this recovery mechanism, see the Manage Replica - Shared Data documentation. References • sql-reference/sql-statements/cluster-management/tablet_replica/ADMIN_REPAIR.md • administration/management/resource_management/Replica.md • faq/shared_data_faq.md
a
@Rocky -
Copy code
Repair cloud native materialized view is not supported
r
It appears that while 
ADMIN REPAIR TABLE
was introduced in version 4.1 to handle missing files in shared-data clusters, it currently has a limitation where it does not support direct repair of Materialized View (MV) objects. To clear the 
404 Not Found
error and the stuck 
PublishVersionDaemon
task for a materialized view, follow these steps: 1. Identify the Materialized View Use the 
TableId
or 
PartitionId
from your error log to identify the specific MV and partition. • Partition ID: 
92632
• Table ID (from S3 path): 
92618
Run this query to find the MV name:
Copy code
sql
SELECT db_name, table_name, partition_name 
FROM information_schema.partitions_meta 
WHERE table_id = 92618;
2. Resolution for Asynchronous MVs If it is an Asynchronous Materialized View, the simplest way to clear a stuck version or missing file error is to force a full refresh. This will create new data files and metadata, effectively bypassing the broken historical transaction. Force a refresh:
Copy code
sql
REFRESH MATERIALIZED VIEW <mv_name> FORCE;
_Note: If the MV is very large and you only want to refresh the broken partition, use 
REFRESH MATERIALIZED VIEW <mv_name> PARTITIONS (<partition_name>) FORCE;
._ Re-activate the MV: If the MV has become inactive due to the errors, try activating it first:
Copy code
sql
ALTER MATERIALIZED VIEW <mv_name> ACTIVE;
3. Resolution for Synchronous MVs (Rollups) If it is a Synchronous Materialized View (often used for rollups), it is managed as part of the base table's transaction. If 
ADMIN REPAIR
is not supported for it, the recommended path is to drop and recreate it: 1. Drop the MV:
Copy code
sql
   DROP MATERIALIZED VIEW <mv_name>;
   -- Or if it's a rollup on a specific table:
   -- ALTER TABLE <base_table> DROP ROLLUP <rollup_name>;
2. Recreate the MV: Run your original 
CREATE MATERIALIZED VIEW
statement. Why did this happen? The 
404
error for a 
.log
file during 
PublishVersion
usually indicates that the FE is trying to commit a transaction whose redo logs were cleaned up (vacuumed) or lost during the upgrade's transition period. Because MVs have their own refresh and state-tracking logic, the 
ADMIN REPAIR
tool (which rolls back metadata) is not yet integrated with the MV state machine in version 4.1.6. If the above steps do not clear the FE logs, you may need to drop the MV and recreate it to fully purge the stale metadata from the FE memory. References • using_starrocks/async_mv/troubleshooting_asynchronous_materialized_views.md
a
@Rocky - I was successful in clearing publish failures. The only anomalies I observe now are: WARN (connector-trigger-analyze-pool-0|9185) [StatisticExecutor.collectStatistics():598] execute statistics job failed: StatisticsCollectJob{type=SAMPLE, scheduleType=ONCE, db=com.starrocks.catalog.Database@5f5e1b6, table=Table [id=100775506, name=sd_wan_edge_link_metrics_reporting, type=ICEBERG], columnNames=[edgeid, clientid, linkid, serialnumber, address1, hierarchy, host, managementstatus, linkstate, timestamp, edgestate], properties={}} com.starrocks.connector.exception.StarRocksConnectorException: Partition values size 1 not match spec fields size 2 in host=mycorp.com/timestamp_day=2026-10-23 at com.starrocks.statistic.ExternalFullStatisticsCollectJob.generatePartitionPredicatesForIcebergTable(ExternalFullStatisticsCollectJob.java:499) ~[fe-core-4.1.jar:?] at com.starrocks.statistic.ExternalFullStatisticsCollectJob.generatePartitionPredicates(ExternalFullStatisticsCollectJob.java:465) ~[fe-core-4.1.jar:?] at com.starrocks.statistic.ExternalFullStatisticsCollectJob.buildPartitionCTESQL(ExternalFullStatisticsCollectJob.java:399) ~[fe-core-4.1.jar:?] at com.starrocks.statistic.ExternalFullStatisticsCollectJob.buildCollectSQLList(ExternalFullStatisticsCollectJob.java:388) ~[fe-core-4.1.jar:?] at com.starrocks.statistic.ExternalFullStatisticsCollectJob.collect(ExternalFullStatisticsCollectJob.java:222) ~[fe-core-4.1.jar:?] at com.starrocks.statistic.StatisticExecutor.collectStatistics(StatisticExecutor.java:595) ~[fe-core-4.1.jar:?] at com.starrocks.connector.statistics.ConnectorAnalyzeTask.executeAnalyze(ConnectorAnalyzeTask.java:184) ~[fe-core-4.1.jar:?] at com.starrocks.connector.statistics.ConnectorAnalyzeTask.run(ConnectorAnalyzeTask.java:145) ~[fe-core-4.1.jar:?] at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1768) ~[?:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[?:?] at java.lang.Thread.run(Thread.java:840) ~[?:?]
r
The error 
Partition values size 1 not match spec fields size 2
occurs when StarRocks' statistics collector (
ANALYZE
) attempts to process an Iceberg table that has an inconsistent or evolved partition specification. Specifically, the error indicates that while the StarRocks FE is generating partition predicates for a statistics job, it encountered a partition (likely 
timestamp_day=2026-10-23
as seen in your logs) that provides only one value, but the table's metadata expects two fields according to the partition spec it is currently evaluating. Likely Causes 1. Partition Spec Evolution: The Iceberg table's partition layout was likely changed (e.g., a column was added or dropped from the partition spec). In Iceberg, older data files may still follow an older 
spec_id
. 2. Connector Logic Mismatch: There is a known class of issues in the Iceberg connector where the statistics job fails to correctly reconcile data files from different partition specs (e.g., identity transforms mixed with other transforms). Recommended Troubleshooting & Workarounds 1. Identify the Partition Spec Discrepancy Check if the table has multiple partition specs by querying the Iceberg metadata tables:
Copy code
sql
-- Check for different spec_ids in the manifests
SELECT DISTINCT partition_spec_id FROM iceberg_catalog.db_name.sd_wan_edge_link_metrics_reporting$manifests;

-- Check how partitions are represented
SELECT * FROM iceberg_catalog.db_name.sd_wan_edge_link_metrics_reporting$partitions LIMIT 10;
If you see multiple 
partition_spec_id
values, the table has evolved. 2. Manual Refresh of Statistics The automatic 
SAMPLE
job is failing. You can try to run a manual 
FULL
analyze or a limited analyze on specific columns to see if it bypasses the predicate generation logic that is failing:
Copy code
sql
-- Try analyzing only specific non-partition columns first
ANALYZE TABLE iceberg_catalog.db_name.sd_wan_edge_link_metrics_reporting 
COLUMNS (edgeid, clientid, linkid);
3. Disable Automatic Statistics for this Table (Temporary Fix) If the failing background job is cluttering your logs or consuming resources, you can disable automatic collection for this specific Iceberg table:
Copy code
sql
ALTER TABLE iceberg_catalog.db_name.sd_wan_edge_link_metrics_reporting 
SET ("stats_auto_collect" = "false");
4. Drop Stale Statistics The upgrade from 4.1.3 to 4.1.6 might have left an incompatible statistics state for this external table. Clear them and see if the next run succeeds:
Copy code
sql
DROP STATS iceberg_catalog.db_name.sd_wan_edge_link_metrics_reporting;
Conclusion Since this is a 
StarRocksConnectorException
appearing after an upgrade to 4.1.6, it is likely a regression or a bug in how the 4.1 branch handles Iceberg partition evolution during statistics collection. If the manual 
ANALYZE
continues to fail with the same error, I recommend reaching out to the CelerData support team or opening an issue on the StarRocks GitHub with the output of the metadata queries mentioned in step 1. References • sql-reference/sql-statements/statistics/ANALYZE_TABLE.md • data_source/catalog/iceberg/iceberg_meta_table.md • administration/Statistics.md References • data_source/catalog/iceberg/iceberg_meta_table.md • faq/shared_data_faq.md • knowledge/admin_and_operation/shared_data_compaction.md
a
@Rocky - partition spec querying shows only one spec_id. ANALYZE FULL TABLE gives the same error that automated job is encoutering. DROPPING stats didn't help -error is still similar: Partition values size 1 not match spec fields size 2 in host=mycorp.com/timestamp_day=2026-04-30
r
This error indicates a mismatch between how the Iceberg table's Partition Spec is defined and the actual partition values StarRocks is reading from the Iceberg metadata (manifests) during the statistics collection process. Specifically, StarRocks' statistics engine (
ExternalFullStatisticsCollectJob
) believes the table should have 2 partition fields, but it only found 1 value for the partition 
timestamp_day=2026-10-23
. 1. Root Cause Analysis This is likely a regression or a logic bug in the StarRocks 4.1.6 Iceberg connector. During 
ANALYZE
, StarRocks generates a SQL query to collect stats and tries to append 
WHERE
clauses for each partition. To do this, it maps the values in your storage path (e.g., 
timestamp_day=2026-10-23
) to the fields defined in the Iceberg Partition Spec. If your table is only partitioned by 
day(timestamp)
, there should only be 1 field. The "size 2" suggests StarRocks is incorrectly detecting an additional partition field (perhaps the raw 
timestamp
column or a "void" transform). 2. Immediate Workaround: Disable Auto-Stats To stop the failing background tasks from cluttering your logs and consuming resources, disable automatic statistics collection for this specific Iceberg table:
Copy code
sql
ALTER TABLE iceberg_catalog.sd_wan.sd_wan_edge_link_metrics_reporting 
SET ("stats_auto_collect" = "false");
3. Debugging the Metadata Mismatch To confirm if the issue is in the metadata or StarRocks' interpretation, run the following diagnostic queries: * Check the Partition Spec and Values:
Copy code
sql
    -- See how StarRocks perceives the partition values
    SELECT partition_value, spec_id
    FROM iceberg_catalog.sd_wan.sd_wan_edge_link_metrics_reporting$partitions
    LIMIT 5;
_If 
partition_value
only contains one key-value pair but 
spec_id
points to a spec with two fields, the Iceberg table metadata itself may be inconsistent._ * Verify the Schema as StarRocks sees it:
Copy code
sql
    DESCRIBE iceberg_catalog.sd_wan.sd_wan_edge_link_metrics_reporting;
Check if any columns are unexpectedly marked as partition columns. 4. Forced Metadata Refresh Sometimes the FE's metadata cache becomes stale after an upgrade. Try clearing the Iceberg metadata cache and re-running the analysis:
Copy code
sql
-- Disable cache for the session
SET enable_iceberg_metadata_cache = false;

-- Try a manual analyze again
ANALYZE TABLE iceberg_catalog.sd_wan.sd_wan_edge_link_metrics_reporting;
5. Recommendation If the 
ANALYZE
still fails with 
size 1 not match spec fields size 2
after disabling the cache, this is a bug in the 4.1.6 connector logic. * Recommendation: Since you have already tried 
DROP STATS
and 
ANALYZE FULL
without success, please report this to the CelerData support team or open a GitHub issue. Mention that the 
ExternalFullStatisticsCollectJob
is incorrectly calculating 
specFields.size()
for Iceberg tables using time-unit transforms (like 
day()
). * Version Note: Check if a 4.1.7 or later hotfix is available, as this type of connector mismatch is typically addressed in maintenance releases following a major upgrade. References • data_source/catalog/iceberg/iceberg_meta_table.md • administration/Statistics.md • sql-reference/sql-statements/statistics/ANALYZE_TABLE.md References • data_source/catalog/iceberg/iceberg_meta_table.md • data_source/catalog/iceberg/DDL.md • quick_start/iceberg.md • faq/Others.md
a
@Rocky - partition_value is as follows: {"host":"mycorp.com","timestamp_day":"2025-04-14"}
r
The 
partition_value
output confirms that the Iceberg metadata itself is correct (it correctly identifies two fields: 
host
and 
timestamp_day
). However, the error message and stack trace confirm that the StarRocks statistics collector is failing to parse these values when generating the SQL for the 
ANALYZE
job. Root Cause The issue is caused by the colon (
:
) and slashes (
/
) within your 
host
partition values (e.g., 
<https://mycorp.com>
). When StarRocks performs an 
ANALYZE
on an external table, it internally generates a query to fetch statistics. Part of this process involves mapping the partition names (the string representation like `host=https://mycorp.com/timestamp_day=...`) back into individual column values to build a 
WHERE
clause. In version 4.1.6, the logic in 
ExternalFullStatisticsCollectJob
likely uses a delimiter-based split (splitting by 
/
or 
=
) that does not correctly handle values containing those same characters. * The parser sees 
host=<https://mycorp.com/timestamp_day=>...
* It incorrectly splits at the slashes or colons inside the URL. * The resulting count of "values" does not match the 2 fields defined in your partition spec, throwing the 
size 1 not match spec fields size 2
error. Solutions and Workarounds 1. Disable Automatic Statistics (Immediate Fix) Since the background job will continue to fail and log errors, disable it for this specific table:
Copy code
sql
ALTER TABLE iceberg_catalog.sd_wan.sd_wan_edge_link_metrics_reporting 
SET ("stats_auto_collect" = "false");
2. Manual Statistics for Specific Columns The error occurs when the collector tries to build partition-aware predicates. You can often bypass this by collecting statistics only for the columns you need, which may use a different internal path:
Copy code
sql
-- Try analyzing columns without including the 'host' or 'timestamp_day' columns
ANALYZE TABLE iceberg_catalog.sd_wan.sd_wan_edge_link_metrics_reporting 
COLUMNS (edgeid, clientid, linkid, serialnumber);
3. Use a "Hive-Compatible" Value (Long-term) If possible in your upstream Iceberg producer, avoid using raw URLs as partition values. URL-encoding the values (e.g., 
<http://https%3A%2F%2Fmycorp.com|https%3A%2F%2Fmycorp.com>
) before writing to Iceberg usually prevents these types of parsing errors in downstream consumers like StarRocks. 4. Report as a Regression This is a specific bug in the 
ExternalFullStatisticsCollectJob
for the 4.1.x branch. * Action: I recommend opening an issue on the StarRocks GitHub or contacting CelerData support. * Details to provide: Mention that 
ExternalFullStatisticsCollectJob.generatePartitionPredicatesForIcebergTable
fails when a partition column (in this case 
host
) contains characters used as path delimiters (
/
and 
:
), leading to a partition value size mismatch. Summary Your metadata is healthy, but the Statistics Engine in 4.1.6 has a parsing bug regarding special characters in partition paths. Disabling auto-stats for this table is the best way to stabilize your cluster logs until a hotfix (likely in 4.1.7+) is applied. References • data_source/catalog/iceberg/iceberg_meta_table.md • unloading/Spark_connector.md • sql-reference/information_schema/partitions_meta.md • sql-reference/sql-statements/table_bucket_part_index/SHOW_PARTITIONS.md
Open in Slack
PreviousNext
Powered by