http://starrocks.io logo
Join Slack
Powered by
```I20260716 11:43:35.847576 139759945733696 olap_...
# questions-and-troubleshooting
h
Copy code
I20260716 11:43:35.847576 139759945733696 olap_server.cpp:895] try to clear expired replication snapshots!
I20260716 11:43:35.847816 139763559607616 olap_server.cpp:246] All backgroud threads of storage engine have started.
I20260716 11:43:35.847826 139763559607616 starrocks_be.cpp:122] CN start step 9: storage engine start bg threads successfully
I20260716 11:43:35.855759 139763559607616 <http://starlet_server.cc:77]|starlet_server.cc:77]> Starlet grpc server started on 0.0.0.0:9070
I20260716 11:43:35.855951 139759652017728 <http://starlet.cc:151]|starlet.cc:151]> Empty starmanager address, skip reporting!
I20260716 11:43:35.856005 139763559607616 starrocks_be.cpp:131] CN start step 10: staros worker init successfully
I20260716 11:43:35.856021 139763559607616 backend_base.cpp:78] StarRocksInternalService has started listening port on 9060
I20260716 11:43:35.856291 139763559607616 thrift_server.cpp:384] BackendService has started listening port on 9060
I20260716 11:43:35.856345 139763559607616 starrocks_be.cpp:152] CN start step 11: start thrift server successfully
I20260716 11:43:35.859434 139763559607616 starrocks_be.cpp:189] BRPC server bind to host: 0.0.0.0, port: 8060
I20260716 11:43:35.864062 139763559607616 server.cpp:1181] Server[starrocks::LakeServiceImpl+starrocks::BackendInternalServiceImpl<starrocks::PInternalService>] is serving on port=8060.
I20260716 11:43:35.864074 139763559607616 server.cpp:1184] Check out <http://reporting-s3-v2-cn-1:8060> in web browser.
I20260716 11:43:35.864413 139763559607616 starrocks_be.cpp:195] CN start step 12: start brpc server successfully
I20260716 11:43:35.870130 139763559607616 starrocks_be.cpp:205] CN start step 13: start http server successfully
I20260716 11:43:35.870150 139763559607616 arrow_flight_sql_service.cpp:33] [ARROW] Arrow Flight SQL Server is disabled. You can modify `arrow_flight_port` in `be.conf` to a positive value to enable it.
I20260716 11:43:35.870155 139763559607616 starrocks_be.cpp:216] CN start step 14: start arrow flight sql server successfully
I20260716 11:43:35.870420 139763559607616 thrift_server.cpp:384] heartbeat has started listening port on 9050
I20260716 11:43:35.870432 139763559607616 starrocks_be.cpp:234] CN start step 15: start heartbeat server successfully
I20260716 11:43:35.870437 139763559607616 starrocks_be.cpp:236] CN started successfully
I20260716 11:43:38.537866 139759662507584 <http://service.cc:110]|service.cc:110]> Updated starlet worker, id: (0 -> 1), service: 30e67ab7-0c9f-4714-8f79-329bd1d0776e, group: 0
I20260716 11:43:38.537887 139759662507584 <http://starlet.cc:74]|starlet.cc:74]> Starmgr leader changed from '' to 'reporting-s3-v2-fe-0.reporting-s3-v2-fe-search.unifyapps.svc.cluster.local:6090'
I20260716 11:43:39.032312 139759063119424 heartbeat_server.cpp:78] get heartbeat from FE. host:reporting-s3-v2-fe-0.reporting-s3-v2-fe-search.unifyapps.svc.cluster.local, port:9020, cluster id:960912848, node type:1, run_mode:SHARED_DATA, counter:1
W20260716 11:43:39.032361 139759063119424 heartbeat_server.cpp:222] reporting-s3-v2-cn-1.reporting-s3-v2-cn-search.unifyapps.svc.cluster.local not equal to to backend localhost 100.69.15.34
I20260716 11:43:39.032459 139759063119424 heartbeat_server.cpp:243] resolved from fqdn: reporting-s3-v2-cn-1.reporting-s3-v2-cn-search.unifyapps.svc.cluster.local to ip: 100.69.15.34
I20260716 11:43:39.032606 139759063119424 network_util.cpp:128] ipv6 link local address fe80::3c2d:b6ff:fecb:ed5b is skipped
I20260716 11:43:39.032614 139759063119424 heartbeat_server.cpp:271] update localhost done, the new localhost is reporting-s3-v2-cn-1.reporting-s3-v2-cn-search.unifyapps.svc.cluster.local, new local_ip is 100.69.15.34
I20260716 11:43:39.032620 139759063119424 heartbeat_server.cpp:107] Updating master info: TMasterInfo(network_address=TNetworkAddress(hostname=reporting-s3-v2-fe-0.reporting-s3-v2-fe-search.unifyapps.svc.cluster.local, port=9020), cluster_id=960912848, epoch=1, token=<hidden>, backend_ip=reporting-s3-v2-cn-1.reporting-s3-v2-cn-search.unifyapps.svc.cluster.local, http_port=8030, heartbeat_flags=0, backend_id=10002, min_active_txn_id=2, run_mode=SHARED_DATA, disabled_disks=<null>, decommissioned_disks=<null>, encrypted=0, stop_regular_tablet_report=1, node_type=Compute)
I20260716 11:43:39.842024 139760412530240 tablet_manager.cpp:1060] Report all 0 tablets info. max_tablet_rowset_num:0, tablet_id:0, cost:0ms
/opt/starrocks/cn_entrypoint.sh: line 165:    27 Quit                    $STARROCKS_HOME/bin/start_cn.sh $addition_args
@Rocky we have launched the new strarrocks cluster on cn pod is fialing like this and other is running
r
The logs you provided indicate that the StarRocks Compute Node (CN) starts successfully but terminates with a SIGQUIT (Signal 3) shortly after receiving its first heartbeat from the Frontend (FE). The message 
27 Quit $STARROCKS_HOME/bin/start_cn.sh
means the main CN process crashed or was killed, causing the shell wrapper to exit. Based on the logs, here are the most likely causes and troubleshooting steps: 1. Check for a Core Dump or Stack Trace A "Quit" signal typically results in a core dump. The standard 
.INFO
log you shared doesn't contain the crash details. You must check the following files inside the failing pod to find the actual error: * 
be/log/cn.out
: This often contains the stack trace if the process crashed. * 
be/log/cn.WARNING
: Check for any 
FATAL
or 
ERROR
messages immediately preceding the crash. * 
dmesg
or 
kubectl describe pod
: Check if the kernel killed the process due to an OOM (Out of Memory) event (though OOM is usually Signal 9/SIGKILL). 2. IP/Hostname Mismatch The log shows a warning: 
heartbeat_server.cpp:222] reporting-s3-v2-cn-1... not equal to to backend localhost 100.69.15.34
While the CN tries to update itself to the FQDN provided by the FE, this mismatch can cause issues if: * The CN cannot resolve its own FQDN (
reporting-s3-v2-cn-1...
). * The FE registered the CN using an IP that the CN doesn't recognize as its local address. Action: Ensure 
priority_networks
is correctly configured in your 
cn.conf
(or Helm values) to match the Kubernetes Pod network (e.g., 
100.69.0.0/16
), or ensure that FQDN resolution is working perfectly within the cluster. 3. Local Storage/Cache Directory Issues Even though CNs are stateless in shared-data mode, they require a local directory for metadata and data caching (specified by 
storage_root_path
in 
cn.conf
). * Is the path writable? If the CN cannot initialize the 
storage
directory, it will crash. * Is the disk full? If the local disk or PV is full, the process may exit during the first tablet report/initialization. 4. Shared-Data / S3 Connectivity Since this is a 
SHARED_DATA
cluster, the CN must connect to your object storage (S3/MinIO/etc.) immediately upon starting. * If the CN lacks the necessary credentials (IAM roles, environment variables, or 
cn.conf
settings) or cannot reach the S3 endpoint, it may crash during its first attempt to sync with the storage. * Check for any 
starlet
or 
lake
related errors in the 
cn.out
or 
cn.WARNING
logs. 5. Resource Constraints (OOM) If 
cn-1
is failing while 
cn-0
is running, it’s possible 
cn-1
is scheduled on a node with less available memory. StarRocks is very sensitive to memory; if the process attempts to allocate memory beyond the K8s limit, it will be killed. * Action: Check 
kubectl describe pod <pod_name>
for 
Reason: OOMKilled
. If it's not OOMKilled but just "Completed/Error", it's more likely a software crash. Summary of what to do next: 1. Run 
kubectl logs <pod-name> -c starrocks-cn
(if there's a separate container) or 
kubectl exec <pod-name> -- cat /opt/starrocks/be/log/cn.out
. 2. Verify the 
ALTER SYSTEM ADD COMPUTE NODE
command used to add this node. Ensure the address matches what the pod thinks its hostname/IP is. 3. If you are using the StarRocks Operator or Helm chart, ensure the 
storage_root_path
is mapped to a valid volume. writable storage. References • faq/Deploy_faq.md • deployment/deploy_manually.md • administration/management/graceful_exit.md • developers/build-starrocks/Build_in_docker.md • deployment/upgrade.md
h
fe has already added compute node but it is unable to connect @Rocky
r
Based on the logs and the "unable to connect" status, your StarRocks Compute Node (CN) is encountering a mismatch between its local network identity and the identity expected by the Frontend (FE). It then receives a SIGQUIT (Signal 3) and terminates shortly after its first heartbeat. 1. Identify the Cause of "Quit" (Signal 3) The message 
27 Quit $STARROCKS_HOME/bin/start_cn.sh
indicates the CN process died due to a "Quit" signal. This is usually triggered in one of two ways: * External Termination: A process manager (like Kubernetes liveness probes) or a sidecar container sent the signal because the node didn't become "Ready" in time. * Internal Crash: The process encountered a fatal state (like a port conflict or resolution failure) and called 
abort()
or was signaled. Action: Check the file 
/opt/starrocks/be/log/cn.out
. This file captures 
stdout/stderr
and will contain the actual stack trace or error message (e.g., "Address already in use" or "Failed to initialize storage") that isn't in the 
.INFO
log. 2. Hostname/IP Mismatch (The Heartbeat Warning) The log line: 
W... heartbeat_server.cpp:222] reporting-s3-v2-cn-1... not equal to to backend localhost 100.69.15.34
shows that the FE added the CN using its FQDN, but the CN thinks its local IP is 
100.69.15.34
. When the CN receives this heartbeat, it tries to update its "localhost" to match the FE's expectation. If this update causes the CN to try to bind to a new address or if it cannot resolve its own FQDN, it will fail. How to Fix: * Configure 
priority_networks
: In your 
cn.conf
(or Helm values), set 
priority_networks = 100.69.0.0/16
(matching your pod CIDR). This forces the CN to identify itself by the IP the FE is using. * Consistency in 
ADD COMPUTE NODE
: Ensure the address you used in 
ALTER SYSTEM ADD COMPUTE NODE "address:9050"
matches exactly how the CN sees itself. If using FQDNs, ensure the CN pod can resolve its own FQDN (
reporting-s3-v2-cn-1...
). 3. Shared-Data Mode Requirements Since you are in 
SHARED_DATA
mode, the CN is not completely stateless—it requires a local cache directory. * Check 
storage_root_path
: Ensure this is set in 
cn.conf
and points to a writable directory. Even if it's a CN, it needs this for metadata and local caching. * S3/Object Storage Access: The CN must be able to reach your S3 endpoint. If the CN cannot initialize the 
starlet
worker (the component that talks to S3), it will exit. Check for "starlet" or "lake" errors in 
cn.WARNING
or 
cn.out
. 4. Kubernetes Probe Failures If you are using the StarRocks Operator or Helm chart, Kubernetes might be killing the pod because the 
readinessProbe
is failing. If the CN doesn't respond to the probe (usually on 
be_http_port
8040) because it is busy trying to resolve the hostname mismatch, Kubernetes will restart it. Recommended Troubleshooting Steps: 1. Examine 
cn.out
: 
kubectl exec <pod> -- cat /opt/starrocks/be/log/cn.out
2. Verify Status: Run 
SHOW PROC '/compute_nodes';
in MySQL. Look at the 
ErrMsg
column for that node. 3. Check Resources: Ensure the CN has enough memory. A "Quit" can sometimes be a side effect of aggressive memory management in certain container runtimes. Summary of likely fix: Add 
priority_networks
to your 
cn.conf
and ensure the CN can resolve both its own hostname and the FE' hostname and the FE's hostname and the FE hostname. References • administration/management/BE_configuration.md • quick_start/shared-data.md • faq/operation_maintenance_faq.md • administration/management/FE_configuration.md • deployment/deploy_manually.md
h
Copy code
I20260716 11:51:27.472075 140550128803136 block_cache.cpp:53] init starcache engine, block_size: 262144, disk_spaces: [{ path: /opt/starrocks/cn/storage/datacache, size: 75161927680 }]
I20260716 11:51:27.473340 140549999289920 daemon.cpp:140] Current memory statistics: process(2916928) query_pool(0) load(0) metadata(0) compaction(0) schema_change(0) page_cache(0) update(0) passthrough(0) clone(0) consistency(0) datacache(0) jit(0) replication(0) jemalloc_active(34787328) jemalloc_allocated(33043560) jemalloc_metadata(2917056) jemalloc_rss(44503040)
I20260716 11:51:27.473350 140550128803136 mem_cache.cpp:59] init mem cache with segment lru policy: [35,65], segment_freq_bits: 0, slru_name: default_cache_mem_slru
I20260716 11:51:27.473517 140550128803136 block_file.cpp:60] open block file success, fd: 8, path: /opt/starrocks/cn/storage/datacache/blockfile_0.1784202001974016
I20260716 11:51:27.473543 140550128803136 block_file.cpp:60] open block file success, fd: 9, path: /opt/starrocks/cn/storage/datacache/blockfile_1.1784202001974067
I20260716 11:51:27.473559 140550128803136 block_file.cpp:60] open block file success, fd: 10, path: /opt/starrocks/cn/storage/datacache/blockfile_2.1784202001974108
I20260716 11:51:27.473574 140550128803136 block_file.cpp:60] open block file success, fd: 11, path: /opt/starrocks/cn/storage/datacache/blockfile_3.1784202001974148
I20260716 11:51:27.473593 140550128803136 block_file.cpp:60] open block file success, fd: 12, path: /opt/starrocks/cn/storage/datacache/blockfile_4.1784202001974186
I20260716 11:51:27.473609 140550128803136 block_file.cpp:60] open block file success, fd: 13, path: /opt/starrocks/cn/storage/datacache/blockfile_5.1784202001974218
I20260716 11:51:27.473625 140550128803136 block_file.cpp:60] open block file success, fd: 14, path: /opt/starrocks/cn/storage/datacache/blockfile_6.1784202001974248
I20260716 11:51:27.473783 140550128803136 disk_cache.cpp:60] init disk cache with segment lru policy: [35,65], segment_freq_bits: 0, slru_name: default_cache_disk_0_slru
I20260716 11:51:27.473967 140550128803136 star_cache_impl.cpp:148] init starcache success. block_size: 262144, disk checksum: 0, mem_quota: 0, disk_quota: 75161927680, scheduler threads: 0
I20260716 11:51:27.473973 140549958919744 star_cache_impl.cpp:317] load starcache meta start.
I20260716 11:51:27.474169 140550128803136 exec_env.cpp:525] datacache init successfully
I20260716 11:51:27.474183 140550128803136 exec_env.cpp:301] Set storage page cache size 1391569403
I20260716 11:51:27.474191 140550128803136 exec_env.cpp:387] object cache init successfully
I20260716 11:51:27.474202 140550128803136 exec_env.cpp:394] storage page cache init successfully
I20260716 11:51:27.474209 140550128803136 starrocks_be.cpp:110] CN start step 6: cache env init successfully
I20260716 11:51:27.476534 140549895980608 data_dir.cpp:133] path: /opt/starrocks/cn/storage, hash: -1149596689658567575
I20260716 11:51:27.480134 140549895980608 kv_store.cpp:106] rocksdb write buffer size: 173946175, total memory: 6957847018, disk count: 1
I20260716 11:51:27.516692 140549958919744 star_cache_impl.cpp:365] load starcache meta end, costs 42 ms, async: 1, success: 1
I20260716 11:51:27.519248 140549633734208 data_dir.cpp:277] begin loading tablet from meta /opt/starrocks/cn/storage
I20260716 11:51:27.519359 140549633734208 data_dir.cpp:316] load tablet from meta finished, loaded tablet: 0, error tablet: 0, path: /opt/starrocks/cn/storage duration: 0ms
I20260716 11:51:27.519374 140549633734208 data_dir.cpp:349] begin loading rowset from meta /opt/starrocks/cn/storage
I20260716 11:51:27.519383 140549633734208 data_dir.cpp:432] load rowset from meta finished, data dir: /opt/starrocks/cn/storage error/total: 0/0 duration: 0ms
I20260716 11:51:27.520320 140550128803136 starrocks_be.cpp:113] CN start step 7: storage engine init successfully
I20260716 11:51:27.524725 140548983363136 fragment_mgr.cpp:552] FragmentMgr cancel worker start working.
I20260716 11:51:27.531282 140550128803136 exec_env.cpp:668] [PIPELINE] Exec thread pool: thread_num=4
I20260716 11:51:27.538026 140550128803136 pipeline_executor_set.cpp:135] [WORKGROUP] start executors ([name=com] [num_driver_threads=4] [num_scan_threads=4] [num_connector_scan_threads=32] [cpuids=(0,1,2,3)] [conf=([num_total_cores=4] [num_total_driver_threads=4] [num_total_scan_threads=4] [num_total_connector_scan_threads=32] [enable_bind_cpus=true] [enable_cpu_borrowing=true])])
I20260716 11:51:27.538076 140550128803136 pipeline_executor_set_manager.cpp:116] [WORKGROUP] assign shared executors to workgroup [workgroup=(id:0, name:default_wg, version:0, cpu_weight:4, exclusive_cpu_cores:0, mem_limit:6262062316, concurrency_limit:0, bigquery: (cpu_second_limit:0, mem_limit:0, scan_rows_limit:0), spill_mem_limit_threshold:1)]
it is failing like this simpaly no error logs @Rocky
r
The logs indicate that your Compute Node (CN) is crashing with a SIGQUIT (Signal 3) shortly after receiving its first heartbeat from the Frontend (FE). The message 
27 Quit $STARROCKS_HOME/bin/start_cn.sh
means the process with PID 27 terminated unexpectedly. Since you mentioned the FE is "unable to connect" even though you added the node, this usually points to a network identity mismatch or a resolution failure. 1. The FQDN / IP Mismatch (Most Likely) In your logs, the FE tells the CN its address is: 
reporting-s3-v2-cn-1.reporting-s3-v2-cn-search.unifyapps.svc.cluster.local
But the CN thinks its local IP is: 
100.69.15.34
The CN tries to update its "localhost" to match the FE's string. If the pod cannot resolve its own FQDN (
reporting-s3-v2-cn-1...
) to an IP address, or if it cannot bind to that address, it will crash. How to fix: * Set `priority_networks`: In your 
cn.conf
, add a line to force the CN to use the correct network interface:
Copy code
properties
    priority_networks = 100.69.0.0/16
(Adjust the CIDR to match your Kubernetes pod network). * Verify FQDN resolution: Exec into a running pod (or the failing one if it stays up long enough) and run:
Copy code
bash
    ping reporting-s3-v2-cn-1.reporting-s3-v2-cn-search.unifyapps.svc.cluster.local
If this fails, your Kubernetes DNS/Headless service is not correctly configured for FQDN access. 2. Check for the Actual Crash Trace The 
.INFO
log doesn't show why the process "Quit." The actual error is recorded in the 
cn.out
 file. Run this command to see the error:
Copy code
bash
kubectl exec <pod-name> -- cat /opt/starrocks/cn/log/cn.out
Look for lines like 
Address already in use
, 
Failed to initialize storage
, or a C++ stack trace. 3. Starlet / Shared-Data Initialization Your logs show: 
Empty starmanager address, skip reporting!
. In 
SHARED_DATA
mode, the CN must receive the StarManager (FE) address to function. If the FE hasn't been fully configured for shared-data, or if there is a version mismatch between FE and CN, the CN may crash during this hand-off. * FE Check: Ensure your FE has 
run_mode = shared_data
in 
fe.conf
. * Version Check: Ensure FE and CN are running the exact same StarRocks version. 4. Memory Limits You have a 75GB DataCache configured: 
init starcache engine, ..., size: 75161927680
The CN requires a certain amount of RAM to manage the metadata for a cache this size. If your Kubernetes memory limit is too low, the process will be killed (SIGKILL) or might fail an internal allocation and abort (SIGQUIT). * Action: Try reducing the cache size in 
cn.conf
temporarily or increasing the Pod's memory limits to see if it stabilizes. Recommended Next Steps: 1. Primary Fix: Add 
priority_networks
to 
cn.conf
. 2. Verify: Check 
SHOW PROC '/compute_nodes';
in MySQL and check the 
ErrMsg
column. 3. Logs: Provide the output of 
cat /opt/starrocks/cn/log/cn.out
if the crash persists. References • deployment/shared_data/s3.md • quick_start/shared-data.md • deployment/shared_data/azure.md • deployment/shared_data/hdfs.md • deployment/shared_data/minio.md • deployment/deploy_manually.md
2 Views
Open in Slack
PreviousNext
Powered by