http://starrocks.io logo
Join Slack
Powered by
Hello, I have an issue with column statistics on e...
# questions-and-troubleshooting
z
Hello, I have an issue with column statistics on external catalog tables (we use a shared architecture, and the tables are stored in an object store in Iceberg format). The error says: 
[42000][1064] only found column statistics: {9530: UsrPersonalNumber, 9707: right, 9549: UsrUserId}, but missing statistic of col: 9523: MobilePhone.
I have updated the statistics for that column even full table (multiple times), and they have been successfully updated. I verified this here:
Copy code
SHOW ANALYZE STATUS
and even here
Copy code
default_catalog._statistics_.external_column_statistics
but I still face that error. any ideas what it can be? @Rocky
r
This error typically occurs when the StarRocks Cost-Based Optimizer (CBO) encounters a mismatch between the statistics cached in the Frontend (FE) memory and the actual table schema or the data stored in the metadata tables. Since you've verified the statistics exist in 
default_catalog._statistics_.external_column_statistics
, the issue is likely a stale cache or a column ID mapping mismatch caused by a schema change in your Iceberg table. Here are the recommended steps to resolve this: 1. Synchronize Metadata and Refresh Cache Since you are using an external catalog, StarRocks might be using an outdated mapping of Column Names to internal Column IDs. * Refresh the table metadata:
Copy code
sql
    REFRESH EXTERNAL TABLE your_catalog.your_db.your_table;
* Invalidate and re-collect statistics: Sometimes simply updating statistics doesn't clear a "stuck" cache entry in the FE. You should explicitly drop them first:
Copy code
sql
    DROP STATS your_catalog.your_db.your_table;
    ANALYZE TABLE your_catalog.your_db.your_table;
2. Toggle Iceberg-Specific Metadata Settings Starting from v3.4, StarRocks can attempt to read statistics directly from Iceberg's own metadata files. If this is enabled but the Iceberg manifest is missing stats for that column, it might conflict with the 
ANALYZE
results in StarRocks. * Disable external metadata stats temporarily to see if it fixes the query:
Copy code
sql
    SET enable_get_stats_from_external_metadata = false;
* Check the Iceberg manifest cache setting in your Catalog properties: If 
iceberg_manifest_cache_with_column_statistics
is set to 
true
, try setting it to 
false
in your 
ALTER CATALOG
statement to rule out manifest-level cache issues. 3. Verify Column "Health" Check if the CBO considers the statistics "healthy" or if it sees a version mismatch:
Copy code
sql
SHOW STATS META WHERE `Table` = 'your_table_name';
Look for the 
Healthy
column. If it's low or if the 
UpdateTime
doesn't match your recent 
ANALYZE
run, the FE is not picking up the new data. 4. Check for Known Bugs related to 
DISTINCT
or 
ALTER
 If your query uses 
DISTINCT
, 
GROUP BY
, or follows a recent 
ALTER TABLE
(column addition/rename), this is a known issue (e.g., #63840) where internal slot IDs get desynchronized. * Workaround: As a temporary measure to bypass the broken statistics check, you can disable the requirement for statistics for the current session:
Copy code
sql
    SET enable_stats_check = false; -- Note: This might lead to sub-optimal plans but will allow the query to run.
5. Clear FE Leader Cache (If Persistent) If the above steps fail, the statistics cache might be stuck in the FE Leader's memory. * Restart the FE Leader node: This forces a complete reload of the statistics and metadata from the 
_statistics_
database. * Alternatively, try running the query on a different FE (if you have multiple) to see if the behavior is consistent. Summary Recommendation: Run 
REFRESH EXTERNAL TABLE ...
, then 
DROP STATS ...
, then a fresh 
ANALYZE TABLE ...
. If it still fails, try 
SET enable_get_stats_from_external_metadata = false;
before running your query. References • data_source/catalog/iceberg/iceberg_meta_table.md • using_starrocks/Cost_based_optimizer.md • data_source/feature-support-data-lake-analytics.md • release_notes/release-3.3.md
z
@peter pang hello, may be you can help again 😞 We've tried everything 😞 So, we are using StarRocks 3.5.15 with a shared architecture. We have Iceberg tables located in object storage. When I run 
DROP STATS
on an external table, queries work. After that, if I run a manual/automatic 
ANALYZE
, it still fails.
Copy code
[42000][1064] only found column statistics: {9530: UsrPersonalNumber, 9707: right, 9549: UsrUserId}, but missing statistic of col: 9523: MobilePhone.
P.S. Checked open issue on git #63840 as well but we do not change the schema of this table at all. It is static and not being updated (even there are no new records and snapshots). Here is the full fe log as well:
Copy code
2026-04-21 00:07:49.172+05:00 WARN (starrocks-mysql-nio-pool-6204|4132344) [StmtExecutor.execute():972] execute Exception, sql: {}, /* ApplicationName=DataGrip 2025.2.2 */ select
    channel,
    campaign,
    count(distinct WingsId) as reqs
from bi.v_cc_attribution_report
where fn_today between '2026-03-01' and '2026-03-31'
  and main = 'Direct Communication'
group by 1,2
order by 3 desc
com.starrocks.sql.common.StarRocksPlannerException: only found column statistics: {9530: UsrPersonalNumber, 9707: right, 9549: UsrUserId}, but missing statistic of col: 9523: MobilePhone.
	at com.starrocks.sql.optimizer.statistics.Statistics.getColumnStatistic(Statistics.java:102) ~[starrocks-fe.jar:?]
	at com.starrocks.sql.optimizer.statistics.ExpressionStatisticCalculator$ExpressionStatisticVisitor.visitVariableReference(ExpressionStatisticCalculator.java:84) ~[starrocks-fe.jar:?]
	at com.starrocks.sql.optimizer.statistics.ExpressionStatisticCalculator$ExpressionStatisticVisitor.visitVariableReference(ExpressionStatisticCalculator.java:63) ~[starrocks-fe.jar:?]
	at com.starrocks.sql.optimizer.operator.scalar.ColumnRefOperator.accept(ColumnRefOperator.java:131) ~[starrocks-fe.jar:?]
	at com.starrocks.sql.optimizer.statistics.ExpressionStatisticCalculator$ExpressionStatisticVisitor.lambda$visitCall$0(ExpressionStatisticCalculator.java:186) ~[starrocks-fe.jar:?]
	at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:197) ~[?:?]
	at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1625) ~[?:?]
	at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509) ~[?:?]
	at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499) ~[?:?]
	at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:921) ~[?:?]
	at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) ~[?:?]
	at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:682) ~[?:?]
	at com.starrocks.sql.optimizer.statistics.ExpressionStatisticCalculator$ExpressionStatisticVisitor.visitCall(ExpressionStatisticCalculator.java:186) ~[starrocks-fe.jar:?]
	at com.starrocks.sql.optimizer.statistics.ExpressionStatisticCalculator$ExpressionStatisticVisitor.visitCall(ExpressionStatisticCalculator.java:63) ~[starrocks-fe.jar:?]
	at com.starrocks.sql.optimizer.operator.scalar.CallOperator.accept(CallOperator.java:263) ~[starrocks-fe.jar:?]
	at com.starrocks.sql.optimizer.statistics.ExpressionStatisticCalculator.calculate(ExpressionStatisticCalculator.java:60) ~[starrocks-fe.jar:?]
	at com.starrocks.sql.optimizer.statistics.ExpressionStatisticCalculator.calculate(ExpressionStatisticCalculator.java:52) ~[starrocks-fe.jar:?]
	at com.starrocks.sql.optimizer.statistics.StatisticsCalculator.visitOperator(StatisticsCalculator.java:257) ~[starrocks-fe.jar:?]
	at com.starrocks.sql.optimizer.statistics.StatisticsCalculator.computeIcebergScanNode(StatisticsCalculator.java:567) ~[starrocks-fe.jar:?]
	at com.starrocks.sql.optimizer.statistics.StatisticsCalculator.visitLogicalIcebergScan(StatisticsCalculator.java:511) ~[starrocks-fe.jar:?]
	at com.starrocks.sql.optimizer.statistics.StatisticsCalculator.visitLogicalIcebergScan(StatisticsCalculator.java:187) ~[starrocks-fe.jar:?]
	at com.starrocks.sql.optimizer.operator.logical.LogicalIcebergScanOperator.accept(LogicalIcebergScanOperator.java:135) ~[starrocks-fe.jar:?]
	at com.starrocks.sql.optimizer.statistics.StatisticsCalculator.estimatorStats(StatisticsCalculator.java:210) ~[starrocks-fe.jar:?]
	at com.starrocks.sql.optimizer.task.DeriveStatsTask.execute(DeriveStatsTask.java:63) ~[starrocks-fe.jar:?]
	at com.starrocks.sql.optimizer.task.TaskScheduler.executeTasks(TaskScheduler.java:43) ~[starrocks-fe.jar:?]
	at com.starrocks.sql.optimizer.QueryOptimizer.memoOptimize(QueryOptimizer.java:908) ~[starrocks-fe.jar:?]
	at com.starrocks.sql.optimizer.QueryOptimizer.optimizeByCost(QueryOptimizer.java:260) ~[starrocks-fe.jar:?]
	at com.starrocks.sql.optimizer.QueryOptimizer.optimize(QueryOptimizer.java:199) ~[starrocks-fe.jar:?]
	at com.starrocks.sql.StatementPlanner.createQueryPlanWithReTry(StatementPlanner.java:394) ~[starrocks-fe.jar:?]
	at com.starrocks.sql.StatementPlanner.plan(StatementPlanner.java:150) ~[starrocks-fe.jar:?]
	at com.starrocks.sql.StatementPlanner.plan(StatementPlanner.java:107) ~[starrocks-fe.jar:?]
	at com.starrocks.qe.StmtExecutor.generateExecPlan(StmtExecutor.java:643) ~[starrocks-fe.jar:?]
	at com.starrocks.qe.StmtExecutor.execute(StmtExecutor.java:749) ~[starrocks-fe.jar:?]
	at com.starrocks.qe.ConnectProcessor.executeQueryAttempt(ConnectProcessor.java:523) ~[starrocks-fe.jar:?]
	at com.starrocks.qe.ConnectProcessor.runWithParserStageRetry(ConnectProcessor.java:421) ~[starrocks-fe.jar:?]
	at com.starrocks.qe.ConnectProcessor.handleQuery(ConnectProcessor.java:358) ~[starrocks-fe.jar:?]
	at com.starrocks.qe.ConnectProcessor.dispatch(ConnectProcessor.java:734) ~[starrocks-fe.jar:?]
	at com.starrocks.qe.ConnectProcessor.processOnce(ConnectProcessor.java:1122) ~[starrocks-fe.jar:?]
	at com.starrocks.mysql.nio.ReadListener.lambda$handleEvent$0(ReadListener.java:71) ~[starrocks-fe.jar:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[?:?]
	at java.lang.Thread.run(Thread.java:840) ~[?:?]
p
Looking at this carefully — you’re hitting something pretty specific, and the pattern (DROP STATS works → re-ANALYZE breaks it again) is a strong signal. Let me walk through what’s actually happening and what to check. What the stack trace tells us The error originates in 
StatisticsCalculator.computeIcebergScanNode
→ 
ExpressionStatisticCalculator.visitVariableReference
. When StarRocks plans the Iceberg scan, it builds a per-scan map of 
ColumnRefOperator → ColumnStatistic
. Your query references 4 columns from the scan, but the loader only put 3 of them into the map. Column 9523 (
MobilePhone
) is silently being dropped during the load step. Two important observations: 1. 
SHOW ANALYZE STATUS
showing success and a row existing in 
_statistics_.external_column_statistics
does not prove the stats row is actually valid/parseable. The INSERT can succeed while the payload is broken. 2. After 
DROP STATS
, the planner falls back to Iceberg manifest/Puffin-file stats (min/max/NDV from the file metadata), which is why queries suddenly work. Re-running 
ANALYZE
re-inserts the broken row and re-breaks it. So this is very likely a bad stats row for MobilePhone specifically, not a schema mismatch and not a cache issue. Check the stats row directly (not via SHOW ANALYZE STATUS) Please run these (replace catalog/db/table accordingly):
Copy code
-- 1. Exact-name lookup, including case variants
SELECT catalog_name, db_name, table_name, column_name,
       row_count, data_size, ndv, null_count,
       length(max) AS max_len, length(min) AS min_len,
       max, min, update_time, collection_size
FROM default_catalog._statistics_.external_column_statistics
WHERE table_name = '<your_iceberg_table>'
  AND lower(column_name) = 'mobilephone';

-- 2. Is the case EXACTLY what Iceberg stores?
SELECT column_name, hex(column_name)
FROM default_catalog._statistics_.external_column_statistics
WHERE table_name = '<your_iceberg_table>';

-- 3. Compare with Iceberg schema:
DESC <catalog>.<db>.<table>;
What to look for: • Case mismatch — Iceberg preserves case; if the stats row says 
mobilephone
but the Iceberg column is 
MobilePhone
, the name-based lookup fails. This is a real issue on external catalog stats. • NULL/empty max/min on a VARCHAR column — the loader can treat unparseable min/max as “no stats” and drop the entry. • row_count = 0 or NULL — same result. • Unexpected character in max/min — phone number columns often contain 
+
, spaces, non-ASCII; if a value trips the serializer, the row gets written but can’t be deserialized. • Duplicate rows — two rows for the same column (e.g., from a partial snapshot) will also cause the loader to mis-key the map. Fast workarounds to unblock the query Any of these should restore service without waiting for a fix: 1. Stop stats from being used for this catalog/table. Drop stats and disable automatic collection on that table only:
Copy code
DROP STATS <catalog>.<db>.<table>;
-- Then disable auto-analyze on external tables or at least this one
ADMIN SET FRONTEND CONFIG("enable_collect_full_statistic" = "false");
Iceberg manifest/Puffin stats will be used instead — usually good enough. 2. Surgical: drop stats on MobilePhone only and see if it survives:
Copy code
DROP STATS <catalog>.<db>.<table>(MobilePhone);
-- Don't re-analyze that column
3. Session-level bypass for the specific query:
Copy code
SET cbo_use_histogram_evaluate_list_partition = false;
SET enable_iceberg_column_statistics = false;
-- Try: SET statistic_collect_iceberg_partition_count_with_full_analyze = false;
(Variable names vary across 3.5.x minor versions — you can list them with 
ADMIN SHOW FRONTEND CONFIG LIKE '%iceberg%stat%';
and 
SHOW VARIABLES LIKE '%stat%';
.) 4. Clear the FE stats cache — stats can be written correctly to disk but a stale in-memory map persists:
Copy code
-- Forces a reload
ADMIN REFRESH EXTERNAL TABLE <catalog>.<db>.<table>;
Or restart the FE leader as a last resort. Things that would change my diagnosis Check the view definition too:
Copy code
SHOW CREATE VIEW bi.v_cc_attribution_report;
If 
MobilePhone
is referenced only inside a CASE/COALESCE/CAST expression in the view, some 3.5.x versions have a bug where the expression rewrite produces a 
ColumnRef
that the stats loader can’t match to the base-table column — the column “exists” in stats but the optimizer is looking for it under a different ColumnRef id. That would match your stack trace exactly (failure inside 
ExpressionStatisticCalculator.visitCall
).
Open in Slack
PreviousNext
Powered by