@Rocky Title: stream_load Error in StarRocks on Kubernetes: 'index -1 out of bounds' Body: I'm encountering an issue while using the stream_load API in StarRocks and would appreciate some help. [Environment] • Infrastructure: Kubernetes • StarRocks Architecture: Compute Node setup • Storage Integration: Iceberg + S3 [What I did] • I created a database and a table in the Internal Catalog. • I attempted to insert data into this table using the stream_load API. [Problem/Symptom] • The API call returns an HTTP 200 OK response. • However, the response payload indicates a failure with the following message: Status: 'Fail', 'message': 'index -1 out of bounds for length 0'

Based on the error message

index -1 out of bounds for length 0

and your specific environment (StarRocks on Kubernetes with Compute Nodes and S3 storage), this issue typically occurs when the Frontend (FE) cannot find any available or "Alive" nodes (Backends or Compute Nodes) to coordinate and execute the load job. In a StarRocks Shared-data (Compute Node) architecture, the FE receives the

stream_load

request and must select an active Compute Node (CN) to act as the coordinator. If the list of available nodes is empty, the selection logic fails with this specific Java exception. Recommended Troubleshooting Steps 1. Verify Node Status Ensure that your Compute Nodes (CNs) have successfully joined the cluster and are marked as "Alive". Run the following SQL command in your StarRocks cluster:

sql
SHOW COMPUTE NODES;
-- Or if you are using BEs in shared-nothing mode:
SHOW BACKENDS;

* If the list is empty: The CNs have not been added to the FE. In a Kubernetes environment, check if the StarRocks Operator has successfully registered the CN pods. If you are deploying manually, you may need to run:

sql
    ALTER SYSTEM ADD COMPUTE NODE "cn_host:9050";

* If

Alive

is `false`: Check the CN logs (

<http://cn.INFO|cn.INFO>

or Pod logs) for connectivity issues, especially whether the CN can reach the FE on its RPC ports. 2. Check Storage Volume Configuration Since you are using a Compute Node setup with S3, you are likely using StarRocks 3.x in Shared-data mode. Internal catalog tables in this mode require a configured Storage Volume. Check if a default storage volume is set and enabled:

sql
SHOW STORAGE VOLUMES;

If no volume is marked as

IsDefault

, or if the table was created without a valid storage location, the FE might fail to initialize the tablet mapping for the load, leading to empty target lists. 3. Kubernetes Network/Hostname Resolution In K8s, FEs and CNs communicate using hostnames or IPs. * Ensure the

FE_HOST

and

CN_HOST

are correctly resolvable within the cluster. * If you are sending the

stream_load

request to the FE, it will try to redirect (HTTP 307) your client to a CN. If the client cannot resolve the CN's internal K8s hostname, the load will fail. However, the "index -1" error usually happens before the redirect, on the FE side, when it fails to pick a target node. 4. Table Metadata Check Verify that the table was created correctly and has tablets assigned. Even in shared-data mode, the FE manages "virtual tablets."

sql
SHOW TABLET FROM <your_table_name>;

If this returns an empty set or an error, the table creation might have failed silently or been corrupted due to the lack of available CNs at the time of creation. Summary of the likely cause The message "length 0" refers to an empty list of available nodes (CNs/BEs), and "index -1" is the result of a failed lookup (search) for a suitable coordinator node within that empty list. Immediate Fix: Check

SHOW COMPUTE NODES;

. If no nodes are listed or none are

Alive

, your stream load will continue to fail with this error. Ensure your CN pods are healthy and have registered with the FE leader. References • faq/shared_data_faq.md • administration/management/FE_configuration.md • deployment/shared_data/feature-support-shared-data.md • quick_start/shared-data.md

@Rocky 1. FE Node Selection: When using the stream_load API, does the FE internally select one of the Compute Nodes within the warehouse to request the execution? 2. NodeCount of 0: I noticed through show warehouses that the NodeCount for the default_warehouse is currently 0. I’d like to clarify if the operation will still function correctly in this state or if there's anything I should be aware of.

To answer your follow-up questions regarding the

stream_load

error and the warehouse state: 1. FE Node Selection Yes, the FE internally selects a node to coordinate the request. When you send a

stream_load

request to the FE (Front End) in a Multi-Warehouse (Shared-data) architecture, the process works as follows: * The FE receives the HTTP request and identifies which Warehouse should handle the job. If no warehouse is explicitly specified in the HTTP header (using

-H "warehouse: <name>"

), it defaults to the

default_warehouse

or the one assigned to your user account. * The FE then looks up the list of available Compute Nodes (CNs) belonging to that warehouse. * It selects one of these CNs to act as the "Coordinator" and redirects your client (via an HTTP 307 Temporary Redirect) to that node to perform the actual data transmission and processing. 2. NodeCount of 0 The operation will NOT function correctly in this state. The error message you received,

index -1 out of bounds for length 0

, is the technical manifestation of a NodeCount of 0. * "length 0" refers to the list of available Compute Nodes in your

default_warehouse

. * "index -1" is the result of the FE trying to pick a node from that empty list. Since there are no nodes in the warehouse, the FE has nowhere to send your data. In StarRocks 3.x (shared-data mode), a warehouse is the compute resource; without at least one active node in the warehouse, you cannot perform loads or run queries. Why is NodeCount 0 and how to fix it? In a Kubernetes environment using the StarRocks Operator, a

NodeCount

of 0 usually indicates one of the following: 1. Scaling Issue: The Compute Node (CN) group in your

StarRocksCluster

CRD might be scaled to 0 replicas, or the pods are failing to start. 2. Registration Failure: The CN pods are running, but they haven't successfully joined the FE. This can happen due to network/firewall issues between the CN and FE (specifically ports 9020 and 9030). 3. Warehouse Configuration: If you are using the Multi-Warehouse feature, you may need to manually add nodes to the warehouse or ensure the warehouse is started. * Check node status:

SHOW COMPUTE NODES;

* Check warehouse status:

SHOW WAREHOUSES;

* If you need to resume/start a warehouse:

ALTER WAREHOUSE default_warehouse RESUME;

(Note: This depends on your specific version and whether "Auto-scaling/Auto-resume" is enabled). Action Plan: Check your Kubernetes pods (

kubectl get pods

). If the CN pods are running and healthy, run

SHOW COMPUTE NODES;

in your SQL client. If that list is empty, you need to troubleshoot why the CNs aren't registering with the FE. If the pods are not running, you need to update your K8s deployment to scale the CNs. References • sql-reference/sql-statements/loading_unloading/STREAM_LOAD.md

@Rocky 1. Permission & Resolution: I’ve confirmed that the NodeCount for the default warehouse is 0, which seems to be causing the "index out of bounds" error. However, since I do not have the administrative permissions to execute commands like ALTER WAREHOUSE or access the Kubernetes cluster, could you advise on how this can be resolved or who can assist with scaling the nodes? 2. Web UI/Dashboard: Is there a web-based dashboard or management console (such as StarRocks Manager) where I can monitor the warehouse status and node count instead of using the CLI? 3. CN Direct Request Error: Interestingly, I also tried sending the stream_load request directly to a Compute Node (CN) to bypass the FE, but I encountered the same 'index out of bounds' error. Could you clarify why this error persists even when the request is sent directly to the CN?

It’s clear now that the NodeCount: 0 is the root cause of your issue. Since you are operating in a shared-data (Compute Node) environment, here is the breakdown of your follow-up questions: 1. Permission & Resolution: Who to contact? Since you do not have administrative permissions, you cannot resolve this yourself. You should reach out to the following teams in your organization: * Database Administrators (DBAs): They have the

cluster_admin

role or

SYSTEM-level OPERATE

privileges required to run

ALTER WAREHOUSE

or

ALTER SYSTEM

commands. * DevOps or Platform Engineering Team: Since StarRocks is running on Kubernetes, they manage the StarRocks Operator. They need to check the

StarRocksCluster

configuration. * The likely fix: They need to ensure the

computeNodes

replicas in the Kubernetes manifest (YAML) is set to a value greater than 0 and that the pods are not in a

CrashLoopBackOff

state. 2. Web UI/Dashboard for Monitoring Yes, there are several ways to monitor the status without using the CLI: * FE Web UI (Built-in): StarRocks FEs have a built-in web interface. By default, it is accessible at

http://<FE_IP>:8030

. You can log in with your database credentials to see the status of Backends, Compute Nodes, and basic cluster info. * StarRocks Manager: This is a more comprehensive, external management platform specifically for StarRocks. If your organization has deployed it, you can manage warehouses, monitor performance, and view node counts via a graphical dashboard. * Grafana Dashboards: Most production StarRocks deployments on Kubernetes use a Prometheus/Grafana stack. There is an official StarRocks Grafana dashboard that visualizes node health, memory usage, and warehouse status. 3. Why does 'Direct to CN' still fail? It might seem counterintuitive that hitting a "live" CN results in a "0 nodes" error, but here is why it happens: * The Role of the Coordinator: When you send a request directly to a CN, that node tries to act as the Coordinator. * Metadata Validation: To start the load, the Coordinator must communicate with the FE (Front End) to plan the job and verify the state of the warehouse. * The FE's Global View: If the FE's metadata reports that the

default_warehouse

has 0 nodes (because the nodes haven't registered correctly or are marked as "Dead" in the FE leader's heartbeats), the FE will return an empty list of available nodes to the Coordinator. * The Logic Fail: The Coordinator then tries to pick a node from this empty list provided by the FE to handle the data segments, leading to the same

index -1 out of bounds for length 0

error. Summary: Even if a CN is physically running, if it is not "Alive" and registered in the FE's metadata (as seen in

SHOW COMPUTE NODES

), it cannot successfully process a

stream_load

job. The registration/heartbeat between the CN and FE is likely broken. References • faq/shared_data_faq.md • loading/SparkLoad.md • sql-reference/sql-statements/cluster-management/nodes_processes/SHOW_COMPUTE_NODES.md • sql-reference/information_schema/warehouse_metrics.md • faq/operation_maintenance_faq.md • faq/resource_isolation_faq.md

@Rocky 1. Manual Node Mapping: If the Compute Nodes (CNs) are actually running and healthy, is there a way to manually map or specify a specific node for the stream_load request to bypass the "NodeCount 0" issue? 2. Fundamental Solution: What is the fundamental way to ensure that the warehouse correctly recognizes active nodes and prevents the NodeCount from dropping to 0 in the future?

To address your questions about manually mapping nodes and finding a fundamental solution for the "NodeCount 0" issue in StarRocks: 1. Manual Node Mapping: Can you bypass the FE? While the

stream_load

API allows you to send requests directly to a Compute Node (CN) IP instead of the FE, it does not bypass the "NodeCount 0" logic. * Why it fails: Even if you hit a CN directly, that CN must act as a "Coordinator." To do its job, the Coordinator contacts the FE to obtain a "Load Plan" (which determines how data is distributed). If the FE's metadata indicates that the warehouse has 0 nodes or that the specific CN is not "Alive," the FE will refuse to generate a plan. * Is there a "Force" parameter? No. There is no HTTP header or property in StarRocks that allows a user to force a load onto a node that the FE considers inactive or non-existent in the warehouse. The FE's metadata is the "source of truth" for the cluster state. 2. Fundamental Solutions: Preventing "NodeCount 0" To ensure your warehouse correctly recognizes active nodes and prevents them from "dropping off," your infrastructure team or DBAs should address these three fundamental areas: A. Network & Heartbeat Stability (The most common cause) In Kubernetes, CNs and FEs communicate via heartbeats. If the heartbeat fails, the FE marks the node as

Alive: false

, and the warehouse count drops. * Port Check: Ensure that the Heartbeat Port (9050 by default) is open and reachable from the FE to the CN pods. * Priority Networks: If your Kubernetes nodes have multiple network interfaces, the CN might be reporting the wrong IP address to the FE. The solution is to set the

priority_networks

parameter in the

cn.conf

(or via the StarRocks Operator) to ensure the CN uses the correct CIDR block for the K8s pod network. * FQDN Resolution: If the cluster is configured to use hostnames (FQDNs) rather than IPs, ensure the FE can resolve the CN pod hostnames. B. StarRocks Operator Health Since you are on Kubernetes, the StarRocks Operator is responsible for automatically running the

ALTER SYSTEM ADD COMPUTE NODE

command when a pod starts. * Operator Logs: If the pods are "Running" but

SHOW COMPUTE NODES

is empty, the Operator might be failing to authenticate with the FE to register the nodes. * Reconcile Errors: Check the logs of the

starrocks-controller

pod. It may show errors like

Access Denied

or

Connection Refused

when trying to update the cluster membership. C. Warehouse "Auto-Suspend" (For 3.x Shared-data) If you are using the Multi-Warehouse feature, check if the warehouse has an Auto-stop property. * If a warehouse is suspended to save costs, the node count will drop to 0. * The Fix: Ensure your DBA has configured the warehouse with

AUTO_RESUME = true

. This allows the warehouse to automatically "wake up" when a new query or load request arrives.

sql
    -- Check warehouse properties
    SHOW CREATE WAREHOUSE default_warehouse;

Summary for your Admin/DevOps Team: Since you don't have permissions, you can provide them with this specific checklist: 1. Run

SHOW COMPUTE NODES;

and check the

ErrMsg

column. It usually contains the exact reason why a node is not "Alive." 2. Verify Pod Connectivity: Can the FE leader ping/telnet to the CN pods on port 9050? 3. Check `priority_networks`: Is the CN reporting an IP that the FE can actually reach? 4. Operator Status: Ensure the StarRocks Operator has the correct credentials to manage the cluster's node membership. References • sql-reference/sql-statements/cluster-management/nodes_processes/SHOW_COMPUTE_NODES.md • deployment/deploy_manually.md