<@U0A71G31CDV> On deploying starrocks (FE+CN), the...
# questions-and-troubleshooting
r
@Rocky On deploying starrocks (FE+CN), the helm chart creates 2 kubernetes service for fe: kube-starrocks-fe-service kube-starrocks-fe-search. Why there are 2 services created and what is the use case of each?
r
When deploying StarRocks using the Helm chart (or the StarRocks Operator), it is normal to see these two services. They serve distinct purposes in a Kubernetes environment: 1.
kube-starrocks-fe-service
This is a Standard Service (typically of type
ClusterIP
or
LoadBalancer
). * Purpose: This is the primary entry point for external and internal clients. It provides a single stable IP address (and optional external IP) to access the FE cluster. * Use Cases: * MySQL Client Access: Connecting to port
9030
for running SQL queries. * Web UI/HTTP API: Accessing the StarRocks FE dashboard or REST APIs via port
8030
. * External Integrations: BI tools and data ingestion pipelines connect to this service. * Behavior: It performs load balancing across all available FE pods. 2.
kube-starrocks-fe-search
This is a Headless Service (its
clusterIP
is set to
None
). * Purpose: It is used for Service Discovery and Pod Identity. In Kubernetes, a Headless Service allows the StatefulSet to assign a unique, stable DNS name to each individual FE pod (e.g.,
fe-0.fe-search
,
fe-1.fe-search
). * Use Cases: * Internal Peer Communication: StarRocks FEs need to communicate with each other directly for metadata replication and leader election (using BDB JE). They use these stable DNS names to maintain a consistent group membership even if a pod restarts and gets a new IP. * BE-to-FE Communication: Backend (BE) or Compute Nodes (CN) use this to discover and communicate with specific FE nodes. * Behavior: It does not load balance; instead, it returns the direct IP addresses of the underlying FE pods when queried. Summary Table Service Name | Type | Key Function | Primary Users
fe-service
| ClusterIP/LoadBalancer | Load balancing client traffic | MySQL clients, BI tools, Users
fe-search
| Headless (None) | Pod identity & peer discovery | Internal FE nodes, BE/CN nodes You will notice a similar pattern for BEs (
be-service
and
be-search
) for the same reasons of providing both a balanced entry point and a discovery mechanism for node-to-node communication. Referencesquick_start/helm.mdadministration/Meta_recovery.mddeployment/deploy_manually.mdquick_start/shared-nothing.md
r
@Rocky Why suddenly I am getting these erros in my client? 1. io.vertx.mysqlclient.MySQLException: {errorMessage=Current connection's compute resource(default_warehouse) is not available:{warehouseId=0, computeResource={warehouseId=0}}, please try again., errorCode=1064, sqlState=HY000} 2. Failed to create db sample_db_name for schemademo motdtlqeObservabilityCounter with io.vertx.sqlclient.ClosedConnectionException: Failed to read any response from the server, the underlying connection may have been lost unexpectedly. 3. io.vertx.mysqlclient.MySQLException: {errorMessage=Warehouse default_warehouse is not available., errorCode=5904, sqlState=42000}
Copy code
SHOW COMPUTE NODES\G;
*************************** 1. row ***************************
        ComputeNodeId: 10023
                   IP: kube-starrocks-cn-0.kube-starrocks-cn-search.plattest03.svc.cluster.local
        HeartbeatPort: 9050
               BePort: 9060
             HttpPort: 8040
             BrpcPort: 8060
        LastStartTime: 2026-04-16 16:51:16
        LastHeartbeat: 2026-04-17 06:38:35
                Alive: true
 SystemDecommissioned: false
ClusterDecommissioned: false
               ErrMsg:
              Version: 4.0.1-cd9df36
             CpuCores: 8
             MemLimit: 12.960GB
    NumRunningQueries: 1
           MemUsedPct: 4.17 %
           CpuUsedPct: 6.6 %
     DataCacheMetrics: Status: Normal, DiskUsage: 1.8GB/31.8GB, MemUsage: 24.9MB/2.5GB
       HasStoragePath: true
           StatusCode: OK
          StarletPort: 9070
             WorkerId: 1
        WarehouseName: default_warehouse
            TabletNum: 396
1 row in set (0.00 sec)

ERROR:
No query specified

Starrocks > SHOW WAREHOUSES\G;
*************************** 1. row ***************************
                 Id: 0
               Name: default_warehouse
              State: AVAILABLE
          NodeCount: 0
CurrentClusterCount: 1
    MaxClusterCount: 1
    StartedClusters: 1
         RunningSql: 0
          QueuedSql: 0
          CreatedOn:
          ResumedOn:
          UpdatedOn:
           Property:
            Comment: An internal warehouse init after FE is ready
1 row in set (0.01 sec)

ERROR:
No query specified
Also, I am able to insert data, crerate new table , select dtaa from the table if I manually ssh in the fe pod and make mysql connection and run the queries.
Copy code
SELECT current_warehouse();
ERROR 1064 (HY000): Getting analyzing error from line 1, column 7 to line 1, column 25. Detail message: No matching function with signature: current_warehouse().
Starrocks > SHOW PROPERTY FOR CURRENT_USER;
ERROR 1064 (HY000): Getting syntax error at line 1, column 18. Detail message: Unexpected input 'CURRENT_USER', the most similar input is {DOUBLE_QUOTED_TEXT, SINGLE_QUOTED_TEXT}.
Also, frontends looks healthy;
Copy code
show frontends\G;
*************************** 1. row ***************************
               Id: 2
             Name: kube-starrocks-fe-1.kube-starrocks-fe-search.plattest03.svc.cluster.local_9010_1773372609823
               IP: kube-starrocks-fe-1.kube-starrocks-fe-search.plattest03.svc.cluster.local
      EditLogPort: 9010
         HttpPort: 8030
        QueryPort: 9030
          RpcPort: 9020
             Role: LEADER
        ClusterId: 1758902270
             Join: true
            Alive: true
ReplayedJournalId: 2209750
    LastHeartbeat: 2026-04-17 06:54:50
         IsHelper: true
           ErrMsg:
        StartTime: 2026-04-17 02:41:50
          Version: 4.0.1-cd9df36
*************************** 2. row ***************************
               Id: 3
             Name: kube-starrocks-fe-2.kube-starrocks-fe-search.plattest03.svc.cluster.local_9010_1773372610386
               IP: kube-starrocks-fe-2.kube-starrocks-fe-search.plattest03.svc.cluster.local
      EditLogPort: 9010
         HttpPort: 8030
        QueryPort: 9030
          RpcPort: 9020
             Role: FOLLOWER
        ClusterId: 1758902270
             Join: true
            Alive: true
ReplayedJournalId: 2209749
    LastHeartbeat: 2026-04-17 06:54:50
         IsHelper: true
           ErrMsg:
        StartTime: 2026-04-17 06:12:34
          Version: 4.0.1-cd9df36
*************************** 3. row ***************************
               Id: 1
             Name: kube-starrocks-fe-0.kube-starrocks-fe-search.plattest03.svc.cluster.local_9010_1773372562608
               IP: kube-starrocks-fe-0.kube-starrocks-fe-search.plattest03.svc.cluster.local
      EditLogPort: 9010
         HttpPort: 8030
        QueryPort: 9030
          RpcPort: 9020
             Role: FOLLOWER
        ClusterId: 1758902270
             Join: true
            Alive: true
ReplayedJournalId: 2209749
    LastHeartbeat: 2026-04-17 06:54:50
         IsHelper: true
           ErrMsg:
        StartTime: 2026-04-17 06:52:15
          Version: 4.0.1-cd9df36
3 rows in set (0.02 sec)

ERROR:
No query specified
r
Hello! I'm Rocky, and I can help you troubleshoot these errors. The errors you are seeing (specifically mentioning
default_warehouse
) indicate that you are likely using StarRocks 3.x or later in a shared-data (cloud-native) architecture or a setup where multi-warehouse support is enabled. The core issue is that your compute resource (Warehouse) is currently offline, suspended, or has no active nodes to process your requests. Here is a breakdown of what each error means and how to fix it: 1. Analysis of the Errors * Error 1 & 3 (
Warehouse default_warehouse is not available
/
errorCode=5904
):
This is a specific StarRocks error indicating that the warehouse named
default_warehouse
is not in a "Running" state. In StarRocks, a warehouse is a cluster of Compute Nodes (CN). If the warehouse is suspended or all its nodes are down, queries cannot be executed. * Error 2 (
ClosedConnectionException: Failed to read any response
):
This happens because the Frontend (FE) node received your request but couldn't find any healthy Compute Nodes to send the task to. As a result, the connection may be dropped or timed out abruptly while the FE tries to resolve the resource issue. * ErrorCode 1064: In StarRocks, this code generally translates to "There is no surviving BE/CN node." 2. Common Causes 1. All Compute Nodes (CN) are Down: If the CN processes have crashed or were stopped, the warehouse becomes unavailable. 2. Warehouse Suspension: If you are using a managed version or a specific configuration that supports "Auto-stop/Auto-resume," the warehouse might have been suspended to save resources and failed to resume. 3. Communication Issues: The FE cannot communicate with the CN nodes due to network issues or heartbeat failures. 4. Resource Exhaustion: If the nodes crashed due to Out-of-Memory (OOM) issues, they might remain in a "Dead" state until restarted. 3. How to Troubleshoot and Fix Please run the following SQL commands in your MySQL client to check the status of your cluster: Step 1: Check Node Status Check if your Compute Nodes are alive:
Copy code
sql
SHOW COMPUTE NODES;
-- Or, if you are using standard Backends:
SHOW BACKENDS;
* What to look for: Check the
Alive
column. If it is
false
for all nodes, your compute layer is down. Check the
ErrMsg
column for specific failure reasons (e.g., heartbeat failure). Step 2: Check Warehouse Status If your version supports the
SHOW WAREHOUSES
command:
Copy code
sql
SHOW WAREHOUSES;
* What to look for: Ensure the
State
of
default_warehouse
is
RUNNING
. If it is
SUSPENDED
, you may need to resume it. Step 3: Check System Logs If the nodes are not alive, you should check the logs on the machines where the Compute Nodes (CN) are installed: * CN Logs:
log/cn.out
or
log/cn.WARNING
* FE Logs:
log/fe.warn
or
log/fe.log
(Look for "warehouse" or "no alive nodes" messages). 4. Immediate Resolution * Restart CN Nodes: If the CN processes have stopped, restart them manually on your worker nodes. * Resume Warehouse: If the warehouse is manually suspended, try running:
Copy code
sql
    ALTER WAREHOUSE default_warehouse RESUME;
(Note: The exact syntax for resuming depends on your StarRocks edition/version). If you recently made changes to your configuration or the cluster was under heavy load, it's possible the nodes hit a memory limit and crashed. Checking the
cn.out
log will be the fastest way to confirm an OOM (Out Of Memory) crash. Referencesfaq/shared_data_faq.mdfaq/resource_isolation_faq.md
The errors you are receiving indicate a discrepancy between your Compute Nodes (CN) and the Warehouse management in your StarRocks cluster. Specifically, while your Compute Node appears "Alive" and healthy in the system, the Warehouse (
default_warehouse
) does not recognize that it has any nodes assigned to it
, resulting in
NodeCount: 0
. When a query is sent, the Frontend (FE) sees a warehouse with zero compute power and rejects the connection. 1. The Root Cause: Discrepancy in Node Assignment Looking at your outputs: *
SHOW COMPUTE NODES
shows 1 node which is
Alive: true
and has
StatusCode: OK
. It even claims to belong to
WarehouseName: default_warehouse
. *
SHOW WAREHOUSES
shows
NodeCount: 0
. This happens when the FE's Warehouse Manager hasn't correctly mapped the registered Compute Node into the active compute cluster of the warehouse. Since StarRocks (especially in shared-data/cloud-native mode) uses Warehouses to schedule work, a
NodeCount
of 0 means there is nowhere to send the query. 2. How to Fix This Option A: Manually Bind the Node to the Warehouse Even if the node says it belongs to the warehouse, try explicitly adding/re-binding it. Run the following SQL:
Copy code
sql
-- Replace <IP> and <HEARTBEAT_PORT> with the values from your SHOW COMPUTE NODES output
-- Based on your output: 'kube-starrocks-cn-0...local' and 9050
ALTER WAREHOUSE default_warehouse ADD COMPUTE NODE "kube-starrocks-cn-0.kube-starrocks-cn-search.plattest03.svc.cluster.local:9050";
Note: In some versions, the port used is the HeartbeatPort (9050). Option B: Resume/Refresh the Warehouse If the warehouse is stuck in a state where it thinks it has no nodes, try a manual resume command to trigger a state refresh:
Copy code
sql
ALTER WAREHOUSE default_warehouse RESUME;
Option C: Check for Blacklisted Nodes Sometimes nodes are automatically blacklisted if they fail heartbeats or have high error rates, which might remove them from the Warehouse count but keep them in the "Compute Nodes" list.
Copy code
sql
SHOW COMPUTE NODE BLACKLIST;
If your node appears there, you may need to wait for it to clear or investigate the
ErrMsg
in the blacklist. 3. Why did "Failed to create db" happen? (Error #2) The
ClosedConnectionException
during a
CREATE DATABASE
or schema operation usually occurs because the FE (Frontend) encountered a fatal error or a timeout while trying to verify the available compute resources to initialize metadata. In a shared-data cluster, even DDL operations may check for warehouse availability. When the FE determines no warehouse is available, it might drop the session, causing the "connection lost unexpectedly" error in your Vert.x client. 4. Summary Checklist 1. Check if
NodeCount
increases:
Run
SHOW WAREHOUSES
again. If
NodeCount
is still
0
, your queries will continue to fail. 2. K8s Operator check: Since you are using Kubernetes (
kube-starrocks-cn-0
), ensure the StarRocks Operator is healthy. If the operator is trying to scale the warehouse but failing, it might leave the FE in this inconsistent state. 3. FE Logs: Check the leader FE's
fe.log
. Look for lines containing
Warehouse
or
no available compute node
. This will tell you exactly why the FE thinks the warehouse is empty despite the node being alive. Immediate workaround: If this is a dev cluster and you need it up now, try restarting the FE Leader. This often forces a reconciliation of the compute node metadata and the warehouse state. Referencessql-reference/sql-functions/utility-functions/current_warehouse.mdsql-reference/information_schema/warehouse_metrics.mdfaq/resource_isolation_faq.md
That you can run queries via the manual MySQL CLI but not from your Vert.x application is a very important clue. This usually points to a session-level or user-specific configuration difference, likely related to how StarRocks 4.0 handles multi-warehouse routing. The "smoking gun" in your provided output is: *
SHOW WAREHOUSES
shows
NodeCount: 0
for
default_warehouse
. *
SHOW COMPUTE NODES
shows 1 node is
Alive
, but for some reason, the Warehouse Manager does not "count" it as a member of the warehouse. Here is why you are seeing this and how to fix it: 1. Why it works in CLI but fails in the App When you connect via the manual CLI (especially if you are using the
root
user), StarRocks might be defaulting you to a different resource group or warehouse, or you are bypassing certain warehouse checks. To verify this, run these two commands in both the CLI (where it works) and try to run them via your App:
Copy code
sql
SELECT current_warehouse();
SHOW PROPERTY FOR CURRENT_USER;
It is likely that your application user has a
default_warehouse
property set to a warehouse that is empty or unavailable, whereas your CLI user is using a different one. 2. The
NodeCount: 0
Issue
In StarRocks 4.0 (and cloud-native 3.x), a warehouse is a logical group. Your output shows the warehouse is
AVAILABLE
, but it has no nodes. Queries require at least one node in the
NodeCount
. Even though
SHOW COMPUTE NODES
lists the warehouse name as
default_warehouse
, there may be a metadata desync. You can try to force the node to be recognized by the warehouse: Try running this command to re-bind the compute node:
Copy code
sql
-- Use the IP and HeartbeatPort from your SHOW COMPUTE NODES output
ALTER WAREHOUSE default_warehouse ADD COMPUTE NODE "kube-starrocks-cn-0.kube-starrocks-cn-search.plattest03.svc.cluster.local:9050";
3. StarRocks 4.0 Specifics You are running 4.0.1. This version introduced more robust multi-warehouse and "Automated Warehouse" features. * Warehouse State: If the state is
AVAILABLE
but
NodeCount
is
0
, it means the warehouse has no active compute clusters. * Error 5904: This specifically triggers when the warehouse exists but cannot find an active cluster of nodes to handle the request. 4. Troubleshooting Steps Step A: Check User Default Warehouse If your application user is not
root
, check if they are restricted to
default_warehouse
. If
default_warehouse
is broken (as shown by
NodeCount: 0
), they will fail.
Copy code
sql
-- Check if the app user is assigned to a specific warehouse
SHOW GRANTS FOR 'your_app_user';
Step B: Check for Resource Group Routing In 4.0, queries are routed based on Resource Groups. If your app user is falling into a resource group mapped to an empty warehouse, it will fail.
Copy code
sql
SELECT * FROM information_schema.resource_group_bindings;
Step C: Force Warehouse in App Connection In your Vert.x connection string or right after connecting, try executing:
Copy code
sql
SET warehouse = 'default_warehouse';
(Or another warehouse name if you create a new one). If this fails with the same error, then
default_warehouse
is definitely the problem. Step D: Restart the FE Leader Since your
SHOW COMPUTE NODES
and
SHOW WAREHOUSES
are showing inconsistent data (one says the node is in the warehouse, the other says the warehouse is empty), the FE Leader's metadata cache is likely out of sync. 1. Identify the FE Leader:
SHOW FRONTENDS;
2. Restart that specific FE pod. This often forces the Warehouse Manager to re-scan the
Alive
compute nodes and correctly populate the
NodeCount
. Summary Recommendation The discrepancy between
NodeCount: 0
and an
Alive
CN is the root cause. Restarting your FE Leader is the fastest way to resolve this metadata inconsistency. If that doesn't work, use the
ALTER WAREHOUSE ... ADD COMPUTE NODE
command to manually re-link them. References
The discrepancy between your CLI (working) and your App (failing), combined with the fact that
SHOW WAREHOUSES
shows
NodeCount: 0
, points to a metadata inconsistency in your Frontend (FE) nodes or a version mismatch. Here is the breakdown of why this is happening and how to fix it: 1. The "NodeCount: 0" Inconsistency Your
SHOW COMPUTE NODES
shows one node is
Alive
and assigned to
default_warehouse
. However,
SHOW WAREHOUSES
shows
NodeCount: 0
. * Result: When your application connects, the FE checks the warehouse, sees 0 nodes, and rejects the query with Error 1064/5904. * Why does the CLI work? You are likely connecting to a different FE pod than your application. In a Kubernetes setup, the Service (LoadBalancer) might be routing your app to an FE with "broken" metadata, while your manual
ssh
into a specific pod hits an FE that happens to be the Leader or have a fresher cache. 2. The "No matching function: current_warehouse()" Mystery The fact that
SELECT current_warehouse()
failed with a "No matching function" error is a major red flag. * In StarRocks 4.0, this function must exist if the warehouse feature is active. * If it doesn't exist, it suggests the FE pod you are currently in is running an older version (e.g., 2.x or 3.x) or is not correctly initialized in
shared_data
mode. * Check this: Run
SHOW FRONTENDS;
and compare the
Version
column for all FE nodes. If they are not all
4.0.1
, you have a version mismatch in your cluster. 3. Troubleshooting & Fixes Step A: Verify FE Versions and Leader Status Run this in your CLI:
Copy code
sql
SHOW FRONTENDS;
Check if all FEs have the same version and who the
Leader
is. If your App is hitting a Follower with stale metadata, it will see
NodeCount: 0
. Step B: Force the Warehouse to recognize the Node Run this command to manually re-bind the compute node to the warehouse. This often "wakes up" the warehouse manager:
Copy code
sql
-- Use the exact IP/DNS and HeartbeatPort from your SHOW COMPUTE NODES output
ALTER WAREHOUSE default_warehouse ADD COMPUTE NODE "kube-starrocks-cn-0.kube-starrocks-cn-search.plattest03.svc.cluster.local:9050";
Step C: Fix the
SHOW PROPERTY
Syntax
To check the properties of your app user (replace
'your_user'
with the actual username):
Copy code
sql
-- The keyword CURRENT_USER isn't supported here; use the literal name
SHOW PROPERTY FOR 'root';
Look for a property like
default_warehouse
. If your app user is forced to a warehouse that has 0 nodes, they will fail even if other users work. Step D: Restart the FE Leader (Recommended) Because
SHOW WAREHOUSES
and
SHOW COMPUTE NODES
are disagreeing, the FE's internal
WarehouseManager
is out of sync. 1. Identify the Leader FE from
SHOW FRONTENDS
. 2. Delete/Restart that FE pod. 3. The new Leader will re-scan the heartbeats from the CN nodes and should correctly update the
NodeCount
to
1
. Why the "ClosedConnectionException"? When the FE determines that a warehouse is unavailable, it may abort the session initialization immediately. Some MySQL clients (like Vert.x) interpret this sudden TCP reset as a
ClosedConnectionException
or "lost response" because the FE closed the socket before sending a standard MySQL error packet. Summary Recommendation: The
NodeCount: 0
is the definitive cause of the "Warehouse not available" error. Restart your FE pods (starting with the Leader) to force a metadata refresh. If you are using the StarRocks Operator on K8s, ensure the
StarRocksWarehouse
custom resource is healthy. Referencessql-reference/sql-functions/utility-functions/current_warehouse.mdfaq/shared_data_faq.mdsql-reference/information_schema/warehouse_metrics.md
This set of errors indicates a metadata inconsistency within your StarRocks Frontend (FE) nodes, specifically regarding the Warehouse Manager. Even though your Compute Node (CN) shows as
Alive
and claims to belong to
default_warehouse
, the Warehouse Manager reports a
NodeCount: 0
. When your application attempts to connect, the FE checks the warehouse state, finds no active compute resources, and rejects the session with errors 1064 and 5904. 1. Why the errors are happening * Error 1064/5904: These are "Warehouse not available" errors. Because
NodeCount
is 0, the FE believes there is no compute cluster available to process your SQL, even though the CN node exists in the system. *
ClosedConnectionException
(Error #2):
When the FE determines a warehouse is unavailable during session initialization, it often closes the connection immediately. Your Vert.x client sees this as an unexpected connection loss. * CLI vs. App Difference: You are likely connecting to different FE pods. In Kubernetes, your manual
ssh
hits a specific FE pod (which might have a healthy metadata cache), while your App goes through a Service/LoadBalancer and might hit a "Follower" FE that is out of sync or has failed to initialize the Warehouse Manager. 2. The Root Cause: "No matching function: current_warehouse()" The fact that
SELECT current_warehouse();
failed with "No matching function" in your CLI is the "smoking gun." In StarRocks 4.0, this function is built-in for Warehouse management. If an FE doesn't recognize it, that specific FE has failed to initialize its Warehouse/Cloud-Native components or is running in a mode that doesn't support them (e.g., misconfigured
run_mode
). 3. Recommended Troubleshooting Steps Step 1: Force a Metadata Refresh (Restart FEs) Since your FEs are showing inconsistent data (Nodes are alive but Warehouse count is 0), the internal metadata cache is likely corrupted or out of sync. 1. Identify the Leader FE using
SHOW FRONTENDS;
. 2. Restart the FE Leader pod. This forces a new leader election and a fresh re-scan of all compute nodes. 3. If the issue persists, restart the remaining FE pods one by one. Step 2: Check for Blacklisted Nodes Sometimes StarRocks automatically blacklists nodes if heartbeats are flaky, which removes them from the Warehouse
NodeCount
but keeps them in
SHOW COMPUTE NODES
.
Copy code
sql
SHOW COMPUTE NODE BLACKLIST;
If your node is listed there, check the
ErrMsg
to see why it was removed. Step 3: Manually Re-bind the Node to the Warehouse Try to explicitly tell the Warehouse Manager to use the CN node. Use the IP/DNS and HeartbeatPort (9050) from your
SHOW COMPUTE NODES
output:
Copy code
sql
-- Replace the string with the IP:Port from your specific output
ALTER WAREHOUSE default_warehouse ADD COMPUTE NODE "kube-starrocks-cn-0.kube-starrocks-cn-search.plattest03.svc.cluster.local:9050";
Step 4: Verify the "run_mode" Ensure all FEs are correctly configured for shared-data/cloud-native mode. Check the FE configuration (usually in
fe.conf
or K8s ConfigMap):
Copy code
properties
run_mode = shared_data
If one FE was accidentally started in
shared_nothing
mode, it will not support warehouses or the
current_warehouse()
function. Summary The
NodeCount: 0
in
SHOW WAREHOUSES
is why your app is failing. Restarting your FE pods (starting with the Leader) is the most effective way to resolve this metadata desynchronization. If you use the StarRocks Operator, also check the status of your
StarRocksWarehouse
or
StarRocksCluster
custom resources to ensure the operator isn't trying to scale the nodes down. Referencessql-reference/sql-statements/cluster-management/nodes_processes/SHOW_BACKEND_BLACKLIST.md
r
@Rocky Why suddenly I am getting these erros in my client? 1. io.vertx.mysqlclient.MySQLException: {errorMessage=Current connection's compute resource(default_warehouse) is not available:{warehouseId=0, computeResource={warehouseId=0}}, please try again., errorCode=1064, sqlState=HY000} 2. Failed to create db sample_db_name for schemademo motdtlqeObservabilityCounter with io.vertx.sqlclient.ClosedConnectionException: Failed to read any response from the server, the underlying connection may have been lost unexpectedly. 3. io.vertx.mysqlclient.MySQLException: {errorMessage=Warehouse default_warehouse is not available., errorCode=5904, sqlState=42000}
Copy code
SHOW COMPUTE NODES\G;
*************************** 1. row ***************************
        ComputeNodeId: 10023
                   IP: kube-starrocks-cn-0.kube-starrocks-cn-search.plattest03.svc.cluster.local
        HeartbeatPort: 9050
               BePort: 9060
             HttpPort: 8040
             BrpcPort: 8060
        LastStartTime: 2026-04-16 16:51:16
        LastHeartbeat: 2026-04-17 06:38:35
                Alive: true
 SystemDecommissioned: false
ClusterDecommissioned: false
               ErrMsg:
              Version: 4.0.1-cd9df36
             CpuCores: 8
             MemLimit: 12.960GB
    NumRunningQueries: 1
           MemUsedPct: 4.17 %
           CpuUsedPct: 6.6 %
     DataCacheMetrics: Status: Normal, DiskUsage: 1.8GB/31.8GB, MemUsage: 24.9MB/2.5GB
       HasStoragePath: true
           StatusCode: OK
          StarletPort: 9070
             WorkerId: 1
        WarehouseName: default_warehouse
            TabletNum: 396
1 row in set (0.00 sec)

ERROR:
No query specified

Starrocks > SHOW WAREHOUSES\G;
*************************** 1. row ***************************
                 Id: 0
               Name: default_warehouse
              State: AVAILABLE
          NodeCount: 0
CurrentClusterCount: 1
    MaxClusterCount: 1
    StartedClusters: 1
         RunningSql: 0
          QueuedSql: 0
          CreatedOn:
          ResumedOn:
          UpdatedOn:
           Property:
            Comment: An internal warehouse init after FE is ready
1 row in set (0.01 sec)

ERROR:
No query specified
Also, I am able to insert data, crerate new table , select dtaa from the table if I manually ssh in the fe pod and make mysql connection and run the queries.
Copy code
SELECT current_warehouse();
ERROR 1064 (HY000): Getting analyzing error from line 1, column 7 to line 1, column 25. Detail message: No matching function with signature: current_warehouse().
Starrocks > SHOW PROPERTY FOR CURRENT_USER;
ERROR 1064 (HY000): Getting syntax error at line 1, column 18. Detail message: Unexpected input 'CURRENT_USER', the most similar input is {DOUBLE_QUOTED_TEXT, SINGLE_QUOTED_TEXT}.
Also, frontends looks healthy;
Copy code
show frontends\G;
*************************** 1. row ***************************
               Id: 2
             Name: kube-starrocks-fe-1.kube-starrocks-fe-search.plattest03.svc.cluster.local_9010_1773372609823
               IP: kube-starrocks-fe-1.kube-starrocks-fe-search.plattest03.svc.cluster.local
      EditLogPort: 9010
         HttpPort: 8030
        QueryPort: 9030
          RpcPort: 9020
             Role: LEADER
        ClusterId: 1758902270
             Join: true
            Alive: true
ReplayedJournalId: 2209750
    LastHeartbeat: 2026-04-17 06:54:50
         IsHelper: true
           ErrMsg:
        StartTime: 2026-04-17 02:41:50
          Version: 4.0.1-cd9df36
*************************** 2. row ***************************
               Id: 3
             Name: kube-starrocks-fe-2.kube-starrocks-fe-search.plattest03.svc.cluster.local_9010_1773372610386
               IP: kube-starrocks-fe-2.kube-starrocks-fe-search.plattest03.svc.cluster.local
      EditLogPort: 9010
         HttpPort: 8030
        QueryPort: 9030
          RpcPort: 9020
             Role: FOLLOWER
        ClusterId: 1758902270
             Join: true
            Alive: true
ReplayedJournalId: 2209749
    LastHeartbeat: 2026-04-17 06:54:50
         IsHelper: true
           ErrMsg:
        StartTime: 2026-04-17 06:12:34
          Version: 4.0.1-cd9df36
*************************** 3. row ***************************
               Id: 1
             Name: kube-starrocks-fe-0.kube-starrocks-fe-search.plattest03.svc.cluster.local_9010_1773372562608
               IP: kube-starrocks-fe-0.kube-starrocks-fe-search.plattest03.svc.cluster.local
      EditLogPort: 9010
         HttpPort: 8030
        QueryPort: 9030
          RpcPort: 9020
             Role: FOLLOWER
        ClusterId: 1758902270
             Join: true
            Alive: true
ReplayedJournalId: 2209749
    LastHeartbeat: 2026-04-17 06:54:50
         IsHelper: true
           ErrMsg:
        StartTime: 2026-04-17 06:52:15
          Version: 4.0.1-cd9df36
3 rows in set (0.02 sec)

ERROR:
No query specified
I am also seeing this:
Copy code
ALTER WAREHOUSE default_warehouse ADD COMPUTE NODE "kube-starrocks-cn-0.kube-starrocks-cn-search.plattest03.svc.cluster.local:9050";
ERROR 1064 (HY000): Getting syntax error at line 1, column 38. Detail message: Unexpected input 'COMPUTE', the most similar input is {'CNGROUP'}.
The rolling restart of FE pods also done.
r
These errors (1064, 5904, and the connection loss) all stem from a single underlying issue: your StarRocks cluster's Warehouse metadata is inconsistent. Even though your Compute Node (CN) shows as
Alive
and claims it belongs to
default_warehouse
, the Warehouse Manager reports a
NodeCount: 0
. Because the FE believes the warehouse has zero compute resources, it rejects incoming queries from your application. Here is a breakdown of why this is happening and how to fix it based on your specific logs: 1. The Root Cause: Metadata Inconsistency In StarRocks 4.0 (Shared-data mode), a Warehouse is a logical container for compute clusters (CNGroups). * The Discrepancy: Your
SHOW COMPUTE NODES
sees the node, but
SHOW WAREHOUSES
does not "count" it. This usually happens if the internal "Cluster" or "CNGroup" inside the warehouse failed to initialize or the metadata is out of sync between FE nodes. * CLI vs. App: Your manual CLI is likely connecting to the FE Leader (which might have the correct metadata in memory), while your App is likely being routed (via a K8s Service) to an FE Follower that is out of sync or has failed to initialize its Warehouse Manager. 2. Addressing the Syntax Errors The errors you got when trying to debug are very revealing: *
current_warehouse()
not found:
This suggests the FE pod you connected to hasn't fully loaded the warehouse function library. * `Unexpected input 'COMPUTE', most similar is {'CNGROUP'}`: This is the key clue. In your specific build of 4.0, the warehouse structure expects a
CNGROUP
(Cluster) level. You cannot add a "Compute Node" directly to a warehouse; you must add a cluster/group to the warehouse first. 3. How to Fix This Step A: Restart the FE Leader (Highly Recommended) Because
SHOW WAREHOUSES
and
SHOW COMPUTE NODES
are disagreeing, the FE's internal
WarehouseManager
is stuck. 1. Run
SHOW FRONTENDS;
and identify the LEADER. 2. Restart that FE pod. This forces a new leader election and a fresh re-scan of all compute nodes, which usually resolves the
NodeCount: 0
issue. Step B: Manually Re-link the Warehouse via CNGROUP Since your version's parser suggested
CNGROUP
, try to explicitly create the cluster link inside the warehouse. Run these commands in order:
Copy code
sql
-- 1. Try to add a cluster/group to the warehouse
-- If 'default_cluster' doesn't work, try another name
ALTER WAREHOUSE default_warehouse ADD CNGROUP cluster_0;

-- 2. Check if the NodeCount in SHOW WAREHOUSES changes to 1.
SHOW WAREHOUSES;
Step C: Check for Blacklisted Nodes Sometimes a node is "Alive" but blacklisted from the warehouse due to a temporary heartbeat failure.
Copy code
sql
SHOW COMPUTE NODE BLACKLIST;
If your node is here, it will not be counted in the
NodeCount
. You can clear it by restarting the CN pod. Step D: Fix User Property Syntax Your syntax for checking properties was slightly off (it doesn't support the
CURRENT_USER
keyword). Check if your app user is restricted to the broken warehouse:
Copy code
sql
-- Use the literal username of your application
SHOW PROPERTY FOR 'your_app_user_name';
Look for
default_warehouse
. If it's set to
default_warehouse
and that warehouse is empty, the app will fail even if you can run queries as
root
. Summary of why your App fails but CLI works: Your application (Vert.x) is likely hitting an FE pod where the
WarehouseManager
has failed to initialize (hence why
current_warehouse()
is missing). When it tries to route your query to
default_warehouse
, it sees
NodeCount: 0
and terminates the connection, leading to the
ClosedConnectionException
. The fastest fix is a rolling restart of your FE pods. Referencessql-reference/sql-statements/cluster-management/nodes_processes/SHOW_COMPUTE_NODES.md
The discrepancy between your CLI (which works for some queries) and your App (which fails), combined with the fact that
SHOW WAREHOUSES
shows
NodeCount: 0
, confirms a metadata desynchronization in your StarRocks Frontend (FE) regarding the Warehouse/Cluster architecture of version 4.0. In StarRocks 4.0 (Shared-data), the hierarchy has changed: WarehouseCNGroup (Cluster)Compute Nodes. Even though your CN node is "Alive," the Warehouse
default_warehouse
thinks it contains 0 nodes. Consequently, when your application connects, the FE rejects the session because it sees no available compute resources to handle the request. 1. Why you are seeing these specific errors * ErrorCode 1064/5904: The FE checks the warehouse state during session initialization. Because
NodeCount
is 0, it declares the warehouse "not available." * ClosedConnectionException: When the FE rejects a connection due to warehouse unavailability, it often closes the TCP socket abruptly before the Vert.x client can read a full MySQL response. * "No matching function: current_warehouse()": This is the "smoking gun." In StarRocks 4.0, this function is a built-in part of the warehouse feature. If it's missing, it means the FE pod you are connected to has failed to initialize the Warehouse Manager or is not running in the correct
shared_data
mode. 2. How to fix the "0 NodeCount" and Metadata Issue Since you have already performed a rolling restart and the issue persists, you need to manually repair the linkage between the Compute Node and the Warehouse using the new CNGroup syntax. Step A: Identify the CNGroup name In version 4.0, nodes must belong to a cluster (CNGroup) within the warehouse. Run this to find your cluster name:
Copy code
sql
SHOW CNGROUPS;
-- or if that fails:
SHOW CLUSTERS;
Step B: Manually Bind the Node to the Cluster Your previous attempt failed because you cannot add a
COMPUTE NODE
directly to a
WAREHOUSE
in 4.0; you must add it to a
CNGROUP
. Assuming your CNGroup is named
cluster_0
(check Step A), run:
Copy code
sql
-- Use the IP/Port from your SHOW COMPUTE NODES output
ALTER CNGROUP cluster_0 ADD COMPUTE NODE "kube-starrocks-cn-0.kube-starrocks-cn-search.plattest03.svc.cluster.local:9050";
_Note: If the CNGROUP doesn't exist, create it first: `CREATE CNGROUP cluster_0;`_ Step C: Add the CNGROUP to the Warehouse Ensure the warehouse is actually looking at that cluster:
Copy code
sql
ALTER WAREHOUSE default_warehouse ADD CNGROUP cluster_0;
Now run
SHOW WAREHOUSES;
again. The
NodeCount
should change from 0 to 1. Once it is 1, your application errors will disappear. 3. Fixing the Syntax for Properties Your
SHOW PROPERTY
failed because
CURRENT_USER
is not a valid keyword for that specific statement. Use the literal username:
Copy code
sql
-- Check properties for the user your app uses
SHOW PROPERTY FOR 'root';
Check if the
default_warehouse
property for that user is set correctly. 4. Final Verification If
SELECT current_warehouse();
still fails after the steps above, check your
fe.conf
on all FE pods. Ensure they all have:
Copy code
properties
run_mode = shared_data
If an FE pod was accidentally started in the wrong mode, it will not support warehouses, leading to the "No matching function" error when your App's LoadBalancer routes traffic to that specific "broken" FE. Referencessql-reference/sql-statements/cluster-management/nodes_processes/SHOW_COMPUTE_NODES.mdsql-reference/sql-statements/cluster-management/nodes_processes/ALTER_SYSTEM.mdadministration/management/BE_blacklist.md
r
@Rocky Why suddenly I am getting these erros in my client? 1. io.vertx.mysqlclient.MySQLException: {errorMessage=Current connection's compute resource(default_warehouse) is not available:{warehouseId=0, computeResource={warehouseId=0}}, please try again., errorCode=1064, sqlState=HY000} 2. Failed to create db sample_db_name for schemademo motdtlqeObservabilityCounter with io.vertx.sqlclient.ClosedConnectionException: Failed to read any response from the server, the underlying connection may have been lost unexpectedly. 3. io.vertx.mysqlclient.MySQLException: {errorMessage=Warehouse default_warehouse is not available., errorCode=5904, sqlState=42000}
Copy code
SHOW COMPUTE NODES\G;
*************************** 1. row ***************************
        ComputeNodeId: 10023
                   IP: kube-starrocks-cn-0.kube-starrocks-cn-search.plattest03.svc.cluster.local
        HeartbeatPort: 9050
               BePort: 9060
             HttpPort: 8040
             BrpcPort: 8060
        LastStartTime: 2026-04-16 16:51:16
        LastHeartbeat: 2026-04-17 06:38:35
                Alive: true
 SystemDecommissioned: false
ClusterDecommissioned: false
               ErrMsg:
              Version: 4.0.1-cd9df36
             CpuCores: 8
             MemLimit: 12.960GB
    NumRunningQueries: 1
           MemUsedPct: 4.17 %
           CpuUsedPct: 6.6 %
     DataCacheMetrics: Status: Normal, DiskUsage: 1.8GB/31.8GB, MemUsage: 24.9MB/2.5GB
       HasStoragePath: true
           StatusCode: OK
          StarletPort: 9070
             WorkerId: 1
        WarehouseName: default_warehouse
            TabletNum: 396
1 row in set (0.00 sec)

ERROR:
No query specified

Starrocks > SHOW WAREHOUSES\G;
*************************** 1. row ***************************
                 Id: 0
               Name: default_warehouse
              State: AVAILABLE
          NodeCount: 0
CurrentClusterCount: 1
    MaxClusterCount: 1
    StartedClusters: 1
         RunningSql: 0
          QueuedSql: 0
          CreatedOn:
          ResumedOn:
          UpdatedOn:
           Property:
            Comment: An internal warehouse init after FE is ready
1 row in set (0.01 sec)

ERROR:
No query specified
Also, I am able to insert data, crerate new table , select dtaa from the table if I manually ssh in the fe pod and make mysql connection and run the queries.
Copy code
SELECT current_warehouse();
ERROR 1064 (HY000): Getting analyzing error from line 1, column 7 to line 1, column 25. Detail message: No matching function with signature: current_warehouse().
Starrocks > SHOW PROPERTY FOR CURRENT_USER;
ERROR 1064 (HY000): Getting syntax error at line 1, column 18. Detail message: Unexpected input 'CURRENT_USER', the most similar input is {DOUBLE_QUOTED_TEXT, SINGLE_QUOTED_TEXT}.
Also, frontends looks healthy;
Copy code
show frontends\G;
*************************** 1. row ***************************
               Id: 2
             Name: kube-starrocks-fe-1.kube-starrocks-fe-search.plattest03.svc.cluster.local_9010_1773372609823
               IP: kube-starrocks-fe-1.kube-starrocks-fe-search.plattest03.svc.cluster.local
      EditLogPort: 9010
         HttpPort: 8030
        QueryPort: 9030
          RpcPort: 9020
             Role: LEADER
        ClusterId: 1758902270
             Join: true
            Alive: true
ReplayedJournalId: 2209750
    LastHeartbeat: 2026-04-17 06:54:50
         IsHelper: true
           ErrMsg:
        StartTime: 2026-04-17 02:41:50
          Version: 4.0.1-cd9df36
*************************** 2. row ***************************
               Id: 3
             Name: kube-starrocks-fe-2.kube-starrocks-fe-search.plattest03.svc.cluster.local_9010_1773372610386
               IP: kube-starrocks-fe-2.kube-starrocks-fe-search.plattest03.svc.cluster.local
      EditLogPort: 9010
         HttpPort: 8030
        QueryPort: 9030
          RpcPort: 9020
             Role: FOLLOWER
        ClusterId: 1758902270
             Join: true
            Alive: true
ReplayedJournalId: 2209749
    LastHeartbeat: 2026-04-17 06:54:50
         IsHelper: true
           ErrMsg:
        StartTime: 2026-04-17 06:12:34
          Version: 4.0.1-cd9df36
*************************** 3. row ***************************
               Id: 1
             Name: kube-starrocks-fe-0.kube-starrocks-fe-search.plattest03.svc.cluster.local_9010_1773372562608
               IP: kube-starrocks-fe-0.kube-starrocks-fe-search.plattest03.svc.cluster.local
      EditLogPort: 9010
         HttpPort: 8030
        QueryPort: 9030
          RpcPort: 9020
             Role: FOLLOWER
        ClusterId: 1758902270
             Join: true
            Alive: true
ReplayedJournalId: 2209749
    LastHeartbeat: 2026-04-17 06:54:50
         IsHelper: true
           ErrMsg:
        StartTime: 2026-04-17 06:52:15
          Version: 4.0.1-cd9df36
3 rows in set (0.02 sec)

ERROR:
No query specified
I am also seeing this:
Copy code
ALTER WAREHOUSE default_warehouse ADD COMPUTE NODE "kube-starrocks-cn-0.kube-starrocks-cn-search.plattest03.svc.cluster.local:9050";
ERROR 1064 (HY000): Getting syntax error at line 1, column 38. Detail message: Unexpected input 'COMPUTE', the most similar input is {'CNGROUP'}.
The rolling restart of FE pods also done.
Copy code
SHOW PROPERTY FOR 'sr_admin_user';
+----------------------+-----------------+
| Key                  | Value           |
+----------------------+-----------------+
| max_user_connections | 1024            |
| catalog              | default_catalog |
| database             |                 |
+----------------------+-----------------+
3 rows in set (0.00 sec)
r
thought Based on the gathered information: The errors "Current connection's compute resource... is not available" and "errorCode=5904" occur because
SHOW WAREHOUSES
reports
NodeCount: 0
. The FE rejects sessions because it thinks there are no compute nodes in the warehouse. In StarRocks 4.0, a "Warehouse" contains "CNGroups" (Clusters), and CNGroups contain "Compute Nodes". * The user's
SHOW WAREHOUSES
output shows
StartedClusters: 1
, but
NodeCount: 0
. This is the core inconsistency. * The error message from their attempt to add a node (
Unexpected input 'COMPUTE', the most similar input is {'CNGROUP'}
) confirms they should be dealing with
CNGROUP
.
current_warehouse()
being missing and manual queries working despite
NodeCount: 0
suggests that manual connections might be bypassing the warehouse-based routing (perhaps because the user property isn't set, so it uses a legacy "internal" path). However, the app (client) is hitting the warehouse path and failing. 1. Identify/Create a CNGroup. 2. Add the Compute Node to the CNGroup. 3. Add the CNGroup to the Warehouse. Also, check
run_mode
. It must be
shared_data
. Wait, why would
current_warehouse()
be missing? The documentation says it's in 4.0. If it's missing, it could be that the FE is not in the correct mode OR the function registration failed because the Warehouse Manager couldn't start (likely due to the metadata mismatch). Plan: 1. Explain the 4.0 hierarchy: Warehouse -> CNGROUP -> Compute Node. 2. Provide the exact commands to link the CN to a CNGroup and then the CNGroup to the warehouse. 3. Mention
run_mode
check. 4. Address the
sr_admin_user
properties—the app user might need
SET WAREHOUSE
or the user property
default_warehouse
correctly set once the warehouse is fixed. One detail:
SHOW COMPUTE NODES
shows
WarehouseName: default_warehouse
for the node. This means the node thinks it's in that warehouse. But the FE (Leader) doesn't see it in the
NodeCount
. The user's
ALTER WAREHOUSE default_warehouse ADD COMPUTE NODE ...
failed because of the 4.0 syntax. Correct 4.0 Syntax for adding a node to a warehouse (based on GitHub/Release clues):
Copy code
sql
ALTER CNGROUP <cngroup_name> ADD COMPUTE NODE "<ip>:<heartbeat_port>";
ALTER WAREHOUSE <warehouse_name> ADD CNGROUP <cngroup_name>;
I need to find what the default CNGROUP name is or how to find it.The errors you are encountering (1064, 5904, and connection loss) are caused by a metadata inconsistency in StarRocks 4.0. Specifically, your
SHOW WAREHOUSES
output shows a
NodeCount: 0
, which means the Frontend (FE) believes the warehouse has no compute resources available. When your application attempts to connect, the FE rejects the session because it cannot find an active compute group to handle the SQL. 1. Why this is happening in StarRocks 4.0 In StarRocks 4.0, the architecture for compute resources has changed to a three-tier hierarchy: Warehouse $\rightarrow$ CNGroup (Cluster) $\rightarrow$ Compute Nodes. Your
SHOW COMPUTE NODES
output shows the node is "Alive" and associated with
default_warehouse
, but the Warehouse Manager layer (responsible for
SHOW WAREHOUSES
) is not "counting" it. This is why: * App Fails: The app connection specifically requests or defaults to
default_warehouse
. Since
NodeCount
is 0, the FE throws error 5904 and closes the connection (
ClosedConnectionException
). * Manual CLI Works: When you connect manually, you may be using a path that doesn't strictly enforce the warehouse (e.g., if the user doesn't have a
default_warehouse
property set), allowing it to fall back to the registered CN nodes directly. * Syntax Errors: Your version (
4.0.1-cd9df36
) uses the CNGroup syntax. You cannot add a "Compute Node" directly to a "Warehouse"; you must add it to a "CNGroup" (Cluster) first. 2. How to Fix the "0 NodeCount" Issue
Since the rolling restart didn't sync the metadata, you must manually repair the linkage using the 4.0
CNGROUP
commands. Step 1: Identify the CNGroups Check if a default CNGroup exists:
Copy code
sql
SHOW CNGROUPS;
-- or
SHOW CLUSTERS;
Step 2: Manually Add the Node to a CNGroup If no group exists, create one. Then, add your compute node to that group using the IP and HeartbeatPort (9050) from your
SHOW COMPUTE NODES
output:
Copy code
sql
-- 1. Create a group if one doesn't exist
CREATE CNGROUP cluster_0;

-- 2. Add the CN node to this group
-- Use the exact IP/DNS from your SHOW COMPUTE NODES output
ALTER CNGROUP cluster_0 ADD COMPUTE NODE "kube-starrocks-cn-0.kube-starrocks-cn-search.plattest03.svc.cluster.local:9050";
Step 3: Link the CNGroup to the Warehouse Ensure the warehouse is configured to use that specific group:
Copy code
sql
ALTER WAREHOUSE default_warehouse ADD CNGROUP cluster_0;
After running these, check
SHOW WAREHOUSES;
. The
NodeCount
should change to 1. Once it is 1, your application's
Current connection's compute resource... is not available
error will be resolved. 3. Verification of
run_mode
The fact that
SELECT current_warehouse();
failed with "No matching function" is a major red flag. This function is a core part of the Warehouse feature in 4.0. If it's missing, ensure all FE pods have the correct
run_mode
in their `fe.conf`:
Copy code
properties
run_mode = shared_data
If an FE pod was started in
shared_nothing
mode, it will not support warehouses or the
current_warehouse()
function, and your App will fail whenever the K8s Service routes traffic to that specific FE. 4. Fix User Properties Your previous
SHOW PROPERTY
command failed due to syntax. To ensure your admin user is correctly bound to the warehouse once fixed, run:
Copy code
sql
-- Check properties
SHOW PROPERTY FOR 'sr_admin_user';

-- If default_warehouse is missing, set it:
ALTER USER 'sr_admin_user' SET PROPERTIES ("default_warehouse" = "default_warehouse");
Summary of Action Items: 1. Run
SHOW CNGROUPS;
to see the cluster name. 2. Run
ALTER CNGROUP ... ADD COMPUTE NODE ...;
to bind the alive node. 3. Run
ALTER WAREHOUSE ... ADD CNGROUP ...;
to link the group to the warehouse. 4. Verify
NodeCount
is 1 in
SHOW WAREHOUSES
. Referencessql-reference/sql-statements/cluster-management/nodes_processes/SHOW_COMPUTE_NODES.mdrelease_notes/release-4.0.mddeployment/deploy_manually.md
r
@Rocky Seeing these errors in fe pod logs:
Copy code
com.starrocks.common.AnalysisException: Getting syntax error at line 9, column 0. Detail message: Unexpected input '{', the most similar input is {<EOF>, ';'}.
	at com.starrocks.qe.ConnectProcessor.executeQueryAttempt(ConnectProcessor.java:440) ~[starrocks-fe.jar:?]
	at com.starrocks.qe.ConnectProcessor.runWithParserStageRetry(ConnectProcessor.java:404) ~[starrocks-fe.jar:?]
	at com.starrocks.qe.ConnectProcessor.handleQuery(ConnectProcessor.java:349) ~[starrocks-fe.jar:?]
	at com.starrocks.qe.ConnectProcessor.dispatch(ConnectProcessor.java:703) ~[starrocks-fe.jar:?]
	at com.starrocks.qe.ConnectProcessor.processOnce(ConnectProcessor.java:1050) ~[starrocks-fe.jar:?]
	at com.starrocks.mysql.nio.MySQLReadListener.handleRequest(MySQLReadListener.java:100) ~[starrocks-fe.jar:?]
	at com.starrocks.mysql.nio.MySQLReadListener.lambda$handleEvent$0(MySQLReadListener.java:86) ~[starrocks-fe.jar:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[?:?]
	at java.lang.Thread.run(Thread.java:840) ~[?:?]
r
The
AnalysisException
in your FE logs confirms that StarRocks is receiving a SQL string from your application that is syntactically invalid—specifically, it is finding a
{
character where it doesn't belong (at line 9, column 0). Combined with your previous "Warehouse not available" errors, here is the likely explanation for what is happening: 1. The "Syntax Error" is likely a Failed Recovery Attempt The error message you see in the FE logs:
Unexpected input '{', the most similar input is {<EOF>, ';'}.
...often happens when an application (like one using Vert.x MySQL client) catches an error and accidentally tries to "re-run" the error message itself as a query, or sends a JSON-formatted configuration/telemetry block to the database. Since the error starts at Line 9, it implies the client sent a large block of text. This usually happens with: * Connection Initializers: If your Vert.x setup has a
connectionInitSql
that contains invalid syntax or is trying to use a JSON/Script block. * ORM Metadata: Some frameworks send multi-line discovery queries to check for database features. * Logging/Proxy Issues: If you have a proxy (like ProxySQL) or a sidecar that is injecting metadata. 2. Why
current_warehouse()
is missing (The Critical Clue)
You mentioned
SHOW WAREHOUSES
works, but
SELECT current_warehouse()
returns "No matching function". In StarRocks 4.0, this function is only registered if the Warehouse Manager initializes successfully. If the Warehouse Manager finds a metadata corruption (like your
NodeCount: 0
discrepancy), it may fail to register the built-in warehouse functions. This is why your app is failing while your manual CLI works: The manual CLI is likely not calling
current_warehouse()
or setting session variables that the Vert.x driver is trying to set automatically (like
SET warehouse = '...'
). 3. How to Debug and Fix Step A: Enable the FE Audit Log To see exactly what query is causing the syntax error, look at the FE audit log on the Leader FE: 1. Check the file:
fe/log/fe.audit.log
. 2. Look for the entry corresponding to the
AnalysisException
. It will show the exact SQL string sent by your Vert.x client. 3. If you see a query starting with
{
or containing a JSON blob, you've found the source in your application code. Step B: Force-Repair the Warehouse Metadata You must resolve the
NodeCount: 0
issue to get the Warehouse Manager back into a healthy state. Because a rolling restart didn't work, do the following: 1. Drop and Re-add the CN node from the Group:
Copy code
sql
    -- Replace with your actual group name and CN address
    ALTER CNGROUP cluster_0 DROP COMPUTE NODE "kube-starrocks-cn-0...:9050";
    ALTER CNGROUP cluster_0 ADD COMPUTE NODE "kube-starrocks-cn-0...:9050";
2. Explicitly set the warehouse for the user: Since your
sr_admin_user
has no
default_warehouse
property (shown in your previous output), the Vert.x client might be trying to "guess" one or failing to initialize a session.
Copy code
sql
    ALTER USER 'sr_admin_user' SET PROPERTIES ("default_warehouse" = "default_warehouse");
Step C: Check Vert.x Connection Configuration Ensure your Vert.x MySQL client is not trying to send multiple statements in a single batch if not enabled, and check if you have any
connectionInitSql
defined in your connection pool settings that might contain a
{
. Summary Recommendation: 1. Check
fe.audit.log
to identify the weird query containing
{
. 2. Fix the
NodeCount: 0
by dropping and re-adding the node to the
CNGROUP
. 3. Update the User Property to point to the fixed warehouse. The
current_warehouse()
function will reappear once the Warehouse Manager successfully initializes with a valid
NodeCount > 0
. Referencesadministration/Meta_recovery.md
r
@Rocky I am seeing this:
Copy code
SHOW PROPERTY FOR 'sr_admin_user';
+----------------------+-----------------+
| Key                  | Value           |
+----------------------+-----------------+
| max_user_connections | 1024            |
| catalog              | default_catalog |
| database             |                 |
+----------------------+-----------------+
3 rows in set (0.00 sec)

Starrocks > ALTER USER 'sr_admin_user' SET PROPERTIES ("default_warehouse" = "default_warehouse");
ERROR 1064 (HY000): Unknown user property(default_warehouse)
r
The errors you are seeing in your application, specifically the
Unknown user property(default_warehouse)
and the
No matching function: current_warehouse()
, are due to the fact that you are running StarRocks 4.0.1. The Warehouse functions and the user-level property were finalized in version 4.1. In 4.0.1, the Warehouse logic exists internally (which is why your connection error references
default_warehouse
), but the SQL helper functions and user properties haven't been exposed yet. 1. Why
NodeCount
is 0 (and how to fix it)
The connection error occurs because
SHOW WAREHOUSES
shows
NodeCount: 0
. In StarRocks 4.0+, nodes are not added directly to a warehouse; they must be added to a CNGroup (Cluster), which is then linked to the Warehouse. Even though
SHOW COMPUTE NODES
says
WarehouseName: default_warehouse
, the Warehouse Manager doesn't "see" it because the node hasn't been explicitly assigned to a compute cluster within that warehouse. Run these commands to fix the linkage: 1. Find the name of your existing cluster:
Copy code
sql
    SHOW CNGROUPS;
    -- If no group exists, create one:
    CREATE CNGROUP cluster_0;
2. Manually bind the CN node to that group: Use the IP and HeartbeatPort (9050) from your
SHOW COMPUTE NODES
output:
Copy code
sql
    ALTER CNGROUP cluster_0 ADD COMPUTE NODE "kube-starrocks-cn-0.kube-starrocks-cn-search.plattest03.svc.cluster.local:9050";
3. Ensure the Warehouse is using that group:
Copy code
sql
    ALTER WAREHOUSE default_warehouse ADD CNGROUP cluster_0;
Now run
SHOW WAREHOUSES;
. Once
NodeCount
is 1, your application's "compute resource not available" error (1064/5904) will stop. ────────── 2. Why
current_warehouse()
and
ALTER USER
failed
*
current_warehouse()
: This function was introduced in 4.1. In 4.0.1, it simply doesn't exist yet. *
Unknown user property
: The ability to set
default_warehouse
via
ALTER USER
was also a 4.1 feature. In 4.0.1, the FE automatically routes connections to the
default_warehouse
if no other warehouse is specified in the session. The Fix: You don't need to set the user property. Once you fix the
NodeCount
in Step 1, the default routing will work automatically. ────────── 3. The
Unexpected input '{'
Error in FE Logs
This error means your Vert.x client is sending a raw JSON string or a curly-bracket block as a SQL query. This often happens in two scenarios: 1. Client-side Error Handling: Your application might be catching a database error and accidentally trying to "execute" the error message (which Vert.x formats as a JSON-like string
{errorMessage=...}
) back into the database. 2. Connection Pool Init: Check if your Vert.x
MySQLConnectOptions
or
PoolOptions
has a
connectionInitSql
that contains a JSON block or an invalid template. Recommendation: Check your
fe.audit.log
. It will show the full SQL string that caused the syntax error. You will likely see your app sending a query like
{ "some": "json" }
instead of a valid SQL statement. Summary Checklist: 1. Fix NodeCount: Use
ALTER CNGROUP
to add the node to a group, then link the group to
default_warehouse
. 2. Verify Version: Keep in mind
current_warehouse()
is a 4.1+ feature; don't use it in your app code for 4.0.1. 3. Audit Log: Check
fe/log/fe.audit.log
to see what query is sending the
{
character. Referencessql-reference/sql-statements/cluster-management/nodes_processes/SHOW_COMPUTE_NODES.mdsql-reference/sql-statements/cluster-management/resource_group/SHOW_RESOURCE_GROUP.md
r
@Rocky This also gives the error:
Copy code
SHOW CNGROUPS;
ERROR 1064 (HY000): Getting syntax error at line 1, column 13. Detail message: Unexpected input '<EOF>', the most similar input is {'FROM'}.
r
The errors you are seeing are a result of the Warehouse & CNGroup architecture introduced in StarRocks 4.0. In StarRocks 4.0, compute resources are organized into a three-layer hierarchy: Warehouse $\rightarrow$ CNGroup (Cluster) $\rightarrow$ Compute Nodes. The reason your application is failing with "compute resource is not available" while
SHOW COMPUTE NODES
shows an active node is that the
NodeCount
for your warehouse is
0
. Even though the node is registered with the system, it has not been assigned to a specific CNGroup linked to that warehouse. 1. Fix the
NodeCount: 0
Issue
You need to manually link your compute node to a CNGroup and then link that group to the warehouse. Step A: Find the CNGroup name Since
SHOW CNGROUPS;
failed with a syntax error, use the 4.0 scoped syntax:
Copy code
sql
SHOW CNGROUPS FROM default_warehouse;
* If this returns a name (e.g.,
cluster_0
), use that in the next step. * If it returns nothing, create one:
CREATE CNGROUP group_0;
and then link it:
ALTER WAREHOUSE default_warehouse ADD CNGROUP group_0;
. Step B: Add the Node to the CNGroup Use the IP/FQDN and HeartbeatPort (9050) from your
SHOW COMPUTE NODES
output:
Copy code
sql
-- Syntax for 4.0: ALTER CNGROUP <group_name> ADD "<ip>:<port>"
ALTER CNGROUP cluster_0 ADD "kube-starrocks-cn-0.kube-starrocks-cn-search.plattest03.svc.cluster.local:9050";
After this, run
SHOW WAREHOUSES;
. The
NodeCount
should now be 1. This will resolve the
MySQLException: Current connection's compute resource... is not available
. 2. Why
current_warehouse()
and
ALTER USER
failed
* Missing Functions: In version 4.0.1, the SQL functions like
current_warehouse()
and the user-level property
default_warehouse
were not yet fully exposed or registered in the same way as in version 4.1. * The Syntax Error: Your
ALTER WAREHOUSE ... ADD COMPUTE NODE
failed because the parser in 4.0 expects a
CNGROUP
keyword, confirming you must manage nodes via groups. 3. The
Unexpected input '{'
in FE Logs
The error
AnalysisException: ... Unexpected input '{'
in your FE logs is likely coming from your Vert.x MySQL client. * The Cause: Some client libraries (or JDBC-based drivers) send ODBC escape sequences like
{fn ...}
or attempt to wrap queries in braces. StarRocks does not support this syntax. * The Connection to Warehouses: When the warehouse is "unavailable" (NodeCount 0), Vert.x often catches the exception and, depending on your error-handling logic, might be accidentally logging/sending the error object (which starts with
{errorMessage=...}
) back to the database as a retry query. Summary of Actions 1. Check Groups:
SHOW CNGROUPS FROM default_warehouse;
2. Bind Node:
ALTER CNGROUP <group_name> ADD "<ip>:9050";
3. Verify: Run
SHOW WAREHOUSES;
and ensure
NodeCount
is 1. 4. Audit Log: If the syntax error
{
persists in logs, check
fe.audit.log
on the leader FE to see exactly what query your app is sending. Note: Ensure your FE and CN are both running with
run_mode = shared_data
in their respective configuration files, as Warehouses are exclusive to the shared-data architecture. Referencessql-reference/sql-statements/cluster-management/nodes_processes/SHOW_COMPUTE_NODES.mdsql-reference/sql-statements/cluster-management/nodes_processes/ALTER_SYSTEM.mdsql-reference/sql-statements/cluster-management/resource_group/CREATE_RESOURCE_GROUP.mdsql-reference/System_variable.md
r
@Rocky How to debug this:
Copy code
com.starrocks.common.AnalysisException: Getting syntax error at line 9, column 0. Detail message: Unexpected input '{', the most similar input is {<EOF>, ';'}.
	at com.starrocks.qe.ConnectProcessor.executeQueryAttempt(ConnectProcessor.java:440) ~[starrocks-fe.jar:?]
	at com.starrocks.qe.ConnectProcessor.runWithParserStageRetry(ConnectProcessor.java:404) ~[starrocks-fe.jar:?]
	at com.starrocks.qe.ConnectProcessor.handleQuery(ConnectProcessor.java:349) ~[starrocks-fe.jar:?]
	at com.starrocks.qe.ConnectProcessor.dispatch(ConnectProcessor.java:703) ~[starrocks-fe.jar:?]
	at com.starrocks.qe.ConnectProcessor.processOnce(ConnectProcessor.java:1050) ~[starrocks-fe.jar:?]
	at com.starrocks.mysql.nio.MySQLReadListener.handleRequest(MySQLReadListener.java:100) ~[starrocks-fe.jar:?]
	at com.starrocks.mysql.nio.MySQLReadListener.lambda$handleEvent$0(MySQLReadListener.java:86) ~[starrocks-fe.jar:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[?:?]
	at java.lang.Thread.run(Thread.java:840) ~[?:?]
r
To debug this syntax error, you need to see the exact SQL string that your application is sending to StarRocks. Because the error occurs at Line 9, the input is a multi-line block, which usually indicates it is not a simple query but potentially a metadata check, a JSON block, or an error object being sent as a query. Follow these steps to isolate the cause: 1. Check the
fe.audit.log
(The Source of Truth)
StarRocks records every query it receives in the audit log, including those that fail with a syntax error. * Location:
fe/log/fe.audit.log
(on your Leader FE pod). * How to search: Use
grep
to find entries with the status
ERR
or search for your database user:
Copy code
bash
    grep "sr_admin_user" fe.audit.log | grep "|Err|"
* What to look for: Look at the
Stmt
field in that log entry. It will show the full text that StarRocks tried to parse. Since the error is at line 9, look for a block that starts with several lines of text followed by a
{
. 2. Common Causes for
{
in StarRocks
StarRocks uses a standard SQL parser. It will throw an
AnalysisException
for
{
if it encounters: * ODBC Escape Sequences: Some drivers send sequences like
{fn ...}
or
{guid ...}
. StarRocks does not support these. * Accidental JSON: If your Vert.x application catches an exception (like your previous "Warehouse not available" error) and tries to log that error into the database or "retry" the error message itself, it might send a JSON-formatted Java exception object starting with
{
. * MyBatis/Template markers: If you are using a mapper and a variable like
${column}
is not being replaced, the raw
{
might be reaching the FE. 3. Correlate with the "Warehouse Not Available" Error In your previous logs, you saw:
MySQLException: Current connection's compute resource ... is not available.
The likely chain of events: 1. Your Vert.x client sends a query. 2. StarRocks returns the "Warehouse not available" error (because
NodeCount
is 0). 3. The Vert.x client or your application's error handler catches this, wraps the error into a JSON object, and accidentally attempts to execute that object as a SQL query (common in "execute-and-retry" loops). 4. The FE sees the JSON
{
and throws the syntax error you see now. 4. Specific Vert.x Debugging If you cannot find the query in the audit log, your client might be failing during the Connection Handshake (e.g., trying to set session variables). * Check `connectionInitSql`: Ensure you don't have any JSON or invalid blocks in your connection pool initialization. * Enable Client Logging: Set your Vert.x logger to
DEBUG
for
io.vertx.mysqlclient
to see the packets being sent before they reach the FE. Summary Recommendation 1. Fix the Warehouse first: Run
ALTER CNGROUP ... ADD COMPUTE NODE ...
to get
NodeCount: 1
. 2. Check the Audit Log: Find the entry for
sr_admin_user
to see what the
{
text actually is. 3. Check Error Handling: Ensure your app isn't trying to "run" error messages as SQL queries. If you find the specific SQL block in the audit log and it looks like a legitimate query you wrote, please share the first 10 lines of that query! Referencesadministration/management/logs.mdfaq/Dump_query.mdfaq/Sql_faq.mdknowledge/trouble_shooting/fe_deadlock_problems.md
r
@Rocky How i can replicate this : 1. Your Vert.x client sends a query. 2. StarRocks returns the "Warehouse not available" error (because
NodeCount
is 0).
r
To replicate the "Warehouse not available" error and debug the syntax issue, you need to understand how Warehouses and CNGroups (Compute Node Groups) work in StarRocks 4.0 (Shared-data mode). 1. How to replicate the "Warehouse not available" error In StarRocks 4.0+, queries require a Warehouse to execute. If a Warehouse has no active compute nodes, you get the error:
Current connection's compute resource(default_warehouse) is not available
. Steps to replicate: 1. Ensure you are in Shared-data mode: This feature only exists when
run_mode = shared_data
is set in
fe.conf
. 2. Verify NodeCount is 0: Run
SHOW WAREHOUSES;
. If the
NodeCount
column is
0
, the warehouse is "empty." 3. Ensure no CNs are linked: * Even if
SHOW COMPUTE NODES
shows nodes are "Alive," they must be bound to a CNGroup that is assigned to the Warehouse. * If you have a
default_warehouse
but haven't run
ALTER CNGROUP ... ADD COMPUTE NODE ...
, the warehouse won't have any resources to run your query. 4. Execute any query: Run
SELECT 1;
or
SHOW TABLES;
. Because there are no compute resources assigned to the warehouse, StarRocks will return the error. 2. Why
ALTER USER ... SET PROPERTIES ("default_warehouse"...)
failed
The error
Unknown user property(default_warehouse)
occurs because: * Version Limitation: The ability to set
default_warehouse
as a user property was introduced in StarRocks 4.1. * In version 4.0.x: The system automatically routes all users to the
default_warehouse
. You cannot yet set this at the user level via
ALTER USER
. You only need to ensure the
default_warehouse
has nodes (Step 1). 3. How to debug the Syntax Error:
Unexpected input '{'
The error
AnalysisException: Unexpected input '{'
at Line 9 is almost certainly not a SQL query you wrote. It happens when your application (Vert.x) sends a non-SQL string to the database. The likely "Succession of Failure": 1. Your app sends a query. 2. StarRocks returns the Warehouse Error (because NodeCount is 0). 3. Vert.x/Your Code catches the exception. 4. Error Handling Bug: Your application accidentally takes the exception object (which is a JSON-like string starting with
{
in some frameworks) and tries to "retry" it or log it by sending it back to the database as a SQL command. How to confirm this: Go to your Leader FE and check the audit log:
Copy code
bash
tail -n 100 fe/log/fe.audit.log | grep "|ERR|"
Look for the
Stmt
field in the log. You will likely see a block of text that looks like a Java stack trace or a JSON object (e.g.,
{ "errorMessage": "...", "errorCode": 1064 }
) instead of
SELECT ...
. How to Fix: 1. Fix the Warehouse: Assign your CN nodes to a group and link it to the warehouse:
Copy code
sql
    -- 1. Create a group if one doesn't exist
    CREATE CNGROUP group_0;
    -- 2. Link the group to the warehouse
    ALTER WAREHOUSE default_warehouse ADD CNGROUP group_0;
    -- 3. Add your CN node (use the IP/Port from SHOW COMPUTE NODES)
    ALTER CNGROUP group_0 ADD "10.0.0.1:9050";
2. Check Vert.x code: Ensure your error handling logic isn't trying to execute caught exceptions. 3. Ignore User Property: Don't use
ALTER USER ... SET PROPERTIES ("default_warehouse"...)
until you upgrade to 4.1. For 4.0, focus on making sure the
default_warehouse
itself is healthy. Referencesfaq/Others.mddata_source/catalog/default_catalog.mddeployment/post_deployment_setup.mdsql-reference/sql-statements/account-management/SHOW_PROPERTY.mddata_source/catalog/unified_catalog.md