http://starrocks.io logo
Join Slack
Powered by
<@U0A71G31CDV> I am running version 4.0.10 in K8s ...
# questions-and-troubleshooting
y
@Rocky I am running version 4.0.10 in K8s is shared mode. I found these log files in one of the FE's:
Copy code
2026-05-17 07:33:25.035Z INFO (StarMgrMetaSyncer|50) [StarMgrMetaSyncer.dropTabletAndDeleteShard():277] Failed to get delete tablet response from node:10003, error: A error occurred: errorCode=2001 errorMessag
2026-05-17 07:33:25.035Z WARN (StarMgrMetaSyncer|50) [StarMgrMetaSyncer.dropTabletAndDeleteShard():174] Failed to batch-resolve shard owners for 0 shards, falling back
com.starrocks.common.StarRocksException: com.staros.client.StarClientException: INVALID_ARGUMENT:shard id can not be empty.
    at com.starrocks.lake.StarOSAgent.getAllNodeIdsByShards(StarOSAgent.java:735) ~[starrocks-fe.jar:?]
    at com.starrocks.lake.StarMgrMetaSyncer.dropTabletAndDeleteShard(StarMgrMetaSyncer.java:171) ~[starrocks-fe.jar:?]
    at com.starrocks.lake.StarMgrMetaSyncer.syncTableMetaInternal(StarMgrMetaSyncer.java:618) ~[starrocks-fe.jar:?]
    at com.starrocks.lake.StarMgrMetaSyncer.syncTableMetaAndColocationInfoInternal(StarMgrMetaSyncer.java:667) ~[starrocks-fe.jar:?]
    at com.starrocks.lake.StarMgrMetaSyncer.syncTableMetaAndColocationInfo(StarMgrMetaSyncer.java:523) ~[starrocks-fe.jar:?]
    at com.starrocks.lake.StarMgrMetaSyncer.runAfterCatalogReady(StarMgrMetaSyncer.java:684) ~[starrocks-fe.jar:?]
    at com.starrocks.common.util.FrontendDaemon.runOneCycle(FrontendDaemon.java:78) ~[starrocks-fe.jar:?]
    at com.starrocks.common.util.Daemon.run(Daemon.java:98) ~[starrocks-fe.jar:?]
Caused by: com.staros.client.StarClientException: INVALID_ARGUMENT:shard id can not be empty.
    at com.staros.client.StarClient.getShardInfo(StarClient.java:600) ~[starclient-4.0.4.jar:?]
    at com.starrocks.lake.StarOSAgent.getShardInfo(StarOSAgent.java:975) ~[starrocks-fe.jar:?]
    at com.starrocks.lake.StarOSAgent.getAllNodeIdsByShards(StarOSAgent.java:727) ~[starrocks-fe.jar:?]
    ... 7 more
2026-05-17 07:33:25.038Z INFO (StarMgrMetaSyncer|50) [StarMgrMetaSyncer.dropTabletAndDeleteShard():277] Failed to get delete tablet response from node:10001, error: A error occurred: errorCode=2001 errorMessag
2026-05-17 07:33:25.038Z WARN (StarMgrMetaSyncer|50) [StarMgrMetaSyncer.dropTabletAndDeleteShard():174] Failed to batch-resolve shard owners for 0 shards, falling back
com.starrocks.common.StarRocksException: com.staros.client.StarClientException: INVALID_ARGUMENT:shard id can not be empty.
    at com.starrocks.lake.StarOSAgent.getAllNodeIdsByShards(StarOSAgent.java:735) ~[starrocks-fe.jar:?]
    at com.starrocks.lake.StarMgrMetaSyncer.dropTabletAndDeleteShard(StarMgrMetaSyncer.java:171) ~[starrocks-fe.jar:?]
    at com.starrocks.lake.StarMgrMetaSyncer.syncTableMetaInternal(StarMgrMetaSyncer.java:618) ~[starrocks-fe.jar:?]
    at com.starrocks.lake.StarMgrMetaSyncer.syncTableMetaAndColocationInfoInternal(StarMgrMetaSyncer.java:667) ~[starrocks-fe.jar:?]
    at com.starrocks.lake.StarMgrMetaSyncer.syncTableMetaAndColocationInfo(StarMgrMetaSyncer.java:523) ~[starrocks-fe.jar:?]
    at com.starrocks.lake.StarMgrMetaSyncer.runAfterCatalogReady(StarMgrMetaSyncer.java:684) ~[starrocks-fe.jar:?]
    at com.starrocks.common.util.FrontendDaemon.runOneCycle(FrontendDaemon.java:78) ~[starrocks-fe.jar:?]
    at com.starrocks.common.util.Daemon.run(Daemon.java:98) ~[starrocks-fe.jar:?]
Caused by: com.staros.client.StarClientException: INVALID_ARGUMENT:shard id can not be empty.
    at com.staros.client.StarClient.getShardInfo(StarClient.java:600) ~[starclient-4.0.4.jar:?]
    at com.starrocks.lake.StarOSAgent.getShardInfo(StarOSAgent.java:975) ~[starrocks-fe.jar:?]
    at com.starrocks.lake.StarOSAgent.getAllNodeIdsByShards(StarOSAgent.java:727) ~[starrocks-fe.jar:?]
    ... 7 more
2026-05-17 07:33:25.040Z INFO (StarMgrMetaSyncer|50) [StarMgrMetaSyncer.dropTabletAndDeleteShard():277] Failed to get delete tablet response from node:933025, error: A error occurred: errorCode=2001 errorMessa
2026-05-17 07:33:25.040Z WARN (StarMgrMetaSyncer|50) [StarMgrMetaSyncer.dropTabletAndDeleteShard():174] Failed to batch-resolve shard owners for 0 shards, falling back
com.starrocks.common.StarRocksException: com.staros.client.StarClientException: INVALID_ARGUMENT:shard id can not be empty.
    at com.starrocks.lake.StarOSAgent.getAllNodeIdsByShards(StarOSAgent.java:735) ~[starrocks-fe.jar:?]
    at com.starrocks.lake.StarMgrMetaSyncer.dropTabletAndDeleteShard(StarMgrMetaSyncer.java:171) ~[starrocks-fe.jar:?]
    at com.starrocks.lake.StarMgrMetaSyncer.syncTableMetaInternal(StarMgrMetaSyncer.java:618) ~[starrocks-fe.jar:?]
    at com.starrocks.lake.StarMgrMetaSyncer.syncTableMetaAndColocationInfoInternal(StarMgrMetaSyncer.java:667) ~[starrocks-fe.jar:?]
    at com.starrocks.lake.StarMgrMetaSyncer.syncTableMetaAndColocationInfo(StarMgrMetaSyncer.java:523) ~[starrocks-fe.jar:?]
    at com.starrocks.lake.StarMgrMetaSyncer.runAfterCatalogReady(StarMgrMetaSyncer.java:684) ~[starrocks-fe.jar:?]
    at com.starrocks.common.util.FrontendDaemon.runOneCycle(FrontendDaemon.java:78) ~[starrocks-fe.jar:?]
    at com.starrocks.common.util.Daemon.run(Daemon.java:98) ~[starrocks-fe.jar:?]
Caused by: com.staros.client.StarClientException: INVALID_ARGUMENT:shard id can not be empty.
    at com.staros.client.StarClient.getShardInfo(StarClient.java:600) ~[starclient-4.0.4.jar:?]
    at com.starrocks.lake.StarOSAgent.getShardInfo(StarOSAgent.java:975) ~[starrocks-fe.jar:?]
    at com.starrocks.lake.StarOSAgent.getAllNodeIdsByShards(StarOSAgent.java:727) ~[starrocks-fe.jar:?]
    ... 7 more
In addition I have this in one of the CN nodes:
Copy code
W20260517 07:35:01.611645 140185102915136 lake_connector.cpp:590] failed to find column in schema: source
W20260517 07:35:01.611648 140185102915136 lake_connector.cpp:590] failed to find column in schema: source
W20260517 07:35:02.427773 140184872138304 lake_connector.cpp:506] failed to find column in schema: source
W20260517 07:35:02.427946 140184872138304 lake_connector.cpp:506] failed to find column in schema: source
W20260517 07:35:02.427987 140184872138304 lake_connector.cpp:506] failed to find column in schema: source
W20260517 07:35:02.428088 140184872138304 lake_connector.cpp:506] failed to find column in schema: source
W20260517 07:35:02.428125 140184872138304 lake_connector.cpp:506] failed to find column in schema: source
W20260517 07:35:02.428197 140184872138304 lake_connector.cpp:590] failed to find column in schema: source
W20260517 07:35:02.428276 140184872138304 lake_connector.cpp:590] failed to find column in schema: source
W20260517 07:35:02.428310 140184872138304 lake_connector.cpp:590] failed to find column in schema: source
W20260517 07:35:02.428366 140184872138304 lake_connector.cpp:590] failed to find column in schema: source
I20260517 07:35:09.606992 140187369731648 daemon.cpp:142] Current memory statistics: process(4134816576) query_pool(644631800) load(0) metadata(70623327) compaction(0) schema_change(0) page_cache(2781694572) u
pdate(0) passthrough(0) clone(0) consistency(0) datacache(2781694572) jit(0) replication(0) jemalloc_active(4347817984) jemalloc_allocated(4120938080) jemalloc_metadata(82202304) jemalloc_rss(4822286336)
I20260517 07:35:14.095895 140187283748416 heartbeat_server.cpp:78] get heartbeat from FE. host:kube-starrocks-fe-1.kube-starrocks-fe-search.data-infra.svc.cluster.local, port:9020, cluster id:943196500, node t
ype:1, run_mode:SHARED_DATA, counter:397
I20260517 07:35:15.338704 140185281242688 tablet_sink_sender.cpp:357] Olap table sink statistics. load_id: 019e34dc-544d-71ec-b2ba-15cffe9633af, txn_id: 809163, add chunk time(ms)/wait lock time(ms)/num: {1000
2:(0)(0)(1)} {932947:(0)(0)(1)} {933025:(0)(0)(1)} {10003:(0)(0)(1)} {933009:(0)(0)(1)} {10001:(0)(0)(1)}
I20260517 07:35:15.445039 140185281242688 tablet_sink_sender.cpp:357] Olap table sink statistics. load_id: 019e34dc-54d2-75b7-aa6b-1dbd11d6f95a, txn_id: 809164, add chunk time(ms)/wait lock time(ms)/num: {1000
2:(0)(0)(1)} {932947:(0)(0)(1)} {933025:(0)(0)(1)} {10003:(0)(0)(1)} {933009:(0)(0)(1)} {10001:(0)(0)(1)}
I20260517 07:35:19.099112 140187283748416 heartbeat_server.cpp:107] Updating master info: TMasterInfo(network_address=TNetworkAddress(hostname=kube-starrocks-fe-1.kube-starrocks-fe-search.data-infra.svc.cluste
r.local, port=9020), cluster_id=943196500, epoch=17, token=<hidden>, backend_ip=kube-starrocks-cn-5.kube-starrocks-cn-search.data-infra.svc.cluster.local, http_port=8030, heartbeat_flags=0, backend_id=933009,
min_active_txn_id=809165, run_mode=SHARED_DATA, disabled_disks=<null>, decommissioned_disks=<null>, encrypted=0, stop_regular_tablet_report=1, node_type=Compute)
I20260517 07:35:24.615907 140187369731648 daemon.cpp:142] Current memory statistics: process(3890781720) query_pool(399734056) load(0) metadata(71330751) compaction(0) schema_change(0) page_cache(2781862513) u
pdate(0) passthrough(0) clone(0) consistency(0) datacache(2781862513) jit(0) replication(0) jemalloc_active(4128079872) jemalloc_allocated(3904811992) jemalloc_metadata(82202304) jemalloc_rss(4604489728)
I20260517 07:35:29.085497 140187283748416 heartbeat_server.cpp:107] Updating master info: TMasterInfo(network_address=TNetworkAddress(hostname=kube-starrocks-fe-1.kube-starrocks-fe-search.data-infra.svc.cluste
r.local, port=9020), cluster_id=943196500, epoch=17, token=<hidden>, backend_ip=kube-starrocks-cn-5.kube-starrocks-cn-search.data-infra.svc.cluster.local, http_port=8030, heartbeat_flags=0, backend_id=933009,
min_active_txn_id=809166, run_mode=SHARED_DATA, disabled_disks=<null>, decommissioned_disks=<null>, encrypted=0, stop_regular_tablet_report=1, node_type=Compute)
I20260517 07:35:31.471131 140185281242688 tablet_sink_sender.cpp:357] Olap table sink statistics. load_id: 019e34dc-9375-71e5-ad90-8facd69c5095, txn_id: 809166, add chunk time(ms)/wait lock time(ms)/num: {1000
2:(0)(0)(1)} {932947:(0)(0)(1)} {933025:(0)(0)(1)} {10003:(0)(0)(1)} {933009:(0)(0)(1)} {10001:(0)(0)(1)}
I20260517 07:35:31.522577 140185281242688 tablet_sink_sender.cpp:357] Olap table sink statistics. load_id: 019e34dc-93ab-727b-9c05-cbea1bfef42c, txn_id: 809167, add chunk time(ms)/wait lock time(ms)/num: {1000
2:(0)(0)(1)} {932947:(0)(0)(1)} {933025:(0)(0)(1)} {10003:(0)(0)(1)} {933009:(0)(0)(1)} {10001:(0)(0)(1)}
I20260517 07:35:34.085153 140187283748416 heartbeat_server.cpp:107] Updating master info: TMasterInfo(network_address=TNetworkAddress(hostname=kube-starrocks-fe-1.kube-starrocks-fe-search.data-infra.svc.cluste
r.local, port=9020), cluster_id=943196500, epoch=17, token=<hidden>, backend_ip=kube-starrocks-cn-5.kube-starrocks-cn-search.data-infra.svc.cluster.local, http_port=8030, heartbeat_flags=0, backend_id=933009,
min_active_txn_id=809168, run_mode=SHARED_DATA, disabled_disks=<null>, decommissioned_disks=<null>, encrypted=0, stop_regular_tablet_report=1, node_type=Compute)
I20260517 07:35:39.620061 140187369731648 daemon.cpp:142] Current memory statistics: process(3740569952) query_pool(245727896) load(0) metadata(71786911) compaction(0) schema_change(0) page_cache(2781791831) u
pdate(0) passthrough(0) clone(0) consistency(0) datacache(2781791831) jit(0) replication(0) jemalloc_active(3984863232) jemalloc_allocated(3746698064) jemalloc_metadata(83279552) jemalloc_rss(4272406528)
I20260517 07:35:54.623493 140187369731648 daemon.cpp:142] Current memory statistics: process(3737620336) query_pool(241568360) load(0) metadata(72098927) compaction(0) schema_change(0) page_cache(2781622703) u
pdate(0) passthrough(0) clone(0) consistency(0) datacache(2781622703) jit(0) replication(0) jemalloc_active(3971989504) jemalloc_allocated(3735699264) jemalloc_metadata(84823744) jemalloc_rss(4203208704)
I20260517 07:35:55.803580 140185239283264 query_context.cpp:63] finished query_id:019e34d8-a968-7ec2-b57d-4537cafa2b57 context life time:280699626911 cpu costs:185841252030 peak memusage:822971432 scan_bytes:2
13463336011 spilled bytes:0 cache_hit_ratio:97.2%
What is the problem ?
r
Based on the logs you provided, you are encountering two distinct issues in your StarRocks v4.0.10 shared-data cluster. 1. FE: 
shard id can not be empty
in 
StarMgrMetaSyncer
 This error occurs in the FE's background metadata synchronization daemon (
StarMgrMetaSyncer
). * Cause: The FE is attempting to clean up or synchronize a tablet that lacks a valid 
shardId
in its metadata. In shared-data mode, every StarRocks tablet must be mapped to a shard managed by StarOS (StarMgr). If a tablet record exists in the FE's internal state with a shard ID of 
0
or empty, any call to StarOS (like 
getShardInfo
or 
dropTablet
) will fail with 
INVALID_ARGUMENT
. * Why it happens: This typically indicates metadata inconsistency or "orphaned" tablets. This can happen if: * A table/partition creation failed halfway, leaving behind tablet records that were never fully initialized with StarOS shards. * There is a synchronization lag between the Leader FE and Follower FEs (a known issue in early 4.0.x versions, e.g., issue #71263, where Follower FEs might miss StarMgr journal updates). * Impact: Usually, this is noisy in the logs but may prevent the background thread from successfully cleaning up deleted data. 2. CN: 
failed to find column in schema: source
 This warning in 
lake_connector.cpp
on the CN indicates a schema mismatch during execution. * Cause: The FE has sent a plan to the CN that expects a column named 
source
, but the CN's local version of the tablet schema for that table does not contain it. * Why it happens: * Concurrent Schema Change: If you recently ran an 
ALTER TABLE
to add or rename a column named 
source
, the CN might be trying to process a load or query using an older version of the schema that hasn't been updated yet. * Inconsistent Metadata: If you are using 
INSERT INTO ... FROM FILES()
or a similar load method, there might be a mismatch between the inferred schema and the target table schema. * Internal Metadata Lag: As noted in the release notes for v4.0.10, there was a race condition in shared-data mode where metadata updates (like shard info or schema versions) could be delayed on some nodes. * Context: Your CN logs show 
Olap table sink statistics
for 
txn_id: 809163
. This confirms the error is happening during a data load. Recommended Actions 1. Identify the affected table: Use the transaction ID from your CN logs to find the table being loaded:
Copy code
sql
    SHOW TRANSACTION FROM <database_name> WHERE id = 809163;
Check the schema of this table (
SHOW CREATE TABLE ...
) and verify if the 
source
column exists and if there are any pending or failed schema changes. 2. Check for Metadata Inconsistency: If the 
shard id can not be empty
error persists and you notice disk space is not being reclaimed after 
DROP TABLE
, you may have orphaned tablets. * Check if any FE Follower nodes are significantly behind the Leader. * Run 
ADMIN SHOW REPLICA DISTRIBUTION FROM <table_name>
to see if tablets are correctly assigned. 3. Upgrade/Patch Note: Your version 4.0.10 (released around May 2026) contains a fix (#71263) for "no queryable replica" issues caused by follower FEs missing StarMgr journal replays. Ensure all FE nodes are running this version and are healthy. If the errors started after an upgrade, verify that the 
meta
directory wasn't corrupted or that a downgrade wasn't attempted (which is unsupported between certain 4.0/3.5 versions). 4. Workaround for Column Error: If the 
source
column error is blocking loads, try restarting the affected CN node to force it to clear its local tablet schema cache and pull the latest version from the FE. References • administration/management/FE_configuration.md
2 Views
Open in Slack
PreviousNext
Powered by