@Rocky I am trying to load one small json file that exists in GCS (81.6 KB) I have permissions to it from my starrocks account. The LOAD is being created but then stuck on state:QUEUEING . how can i debug it? what can i do? this is the code:

LOAD LABEL org_id_to_name_load_test_label_2
(
    DATA INFILE("<gs://data-ads-core-forge-prd/bigquery_exports/unity-data-ads-core-prd/druid_lookups/org_id_to_name/*>")
    INTO TABLE org_id_to_name_load_test
    FORMAT AS "json"
 )
 WITH BROKER
 (
 
    "gcp.gcs.use_compute_engine_service_account" = "true"
 )
PROPERTIES
(
    "timeout" = "300"
);

Hi! I'm Rocky. It sounds like your Broker Load job is stuck in the

QUEUEING

state. This state indicates that the load job has been created successfully but is waiting for a scheduling slot to start execution. Even for a tiny file (81.6 KB), a job can remain in

QUEUEING

if the cluster's concurrency limits are reached or if there's a scheduling bottleneck. Here is how you can debug and resolve this: 1. Check Concurrent Load Jobs StarRocks has a limit on how many Broker Load jobs can run at the same time. The default is often as low as 5. Check how many jobs are currently in the

LOADING

state:

sql
SHOW LOAD WHERE STATE = 'LOADING';

If you have 5 or more jobs running, your new job will wait in the queue. You can increase this limit by modifying the FE configuration (requires admin privileges):

sql
ADMIN SET FRONTEND CONFIG ("max_broker_load_job_concurrency" = "20");

2. Check Backend (BE) Status If no Backends are available or "Alive," the FE cannot schedule the task.

sql
SHOW BACKENDS;

Ensure that the

Alive

column is

true

for your nodes. If all BEs are down, jobs will stay in

QUEUEING

or

PENDING

. 3. Check for Transaction Limits Each database has a limit on concurrent transactions (

max_running_txn_num_per_db

, default is 1000). While unlikely to be hit with one file, if you have many small concurrent loads or unfinished transactions, it could cause delays. 4. Check FE Leader Status Only the Leader FE handles load scheduling. Ensure you are connected to the Leader or that the cluster has a healthy Leader:

sql
SHOW FRONTENDS;

Check the

IsMaster

(or

Role

) column. 5. Inspect the Load Job Details Even though it's queueing, sometimes the

Etc

or

ErrorMsg

column in

SHOW LOAD

provides hints if it attempted to start and failed back into the queue (though this usually leads to

CANCELLED

).

sql
SHOW LOAD WHERE LABEL = 'org_id_to_name_load_test_label_2'\G

6. Resource Group Constraints If you are using Resource Groups, the load might be queued because the resource group has reached its CPU or concurrency limit. You can check resource usage with:

sql
SHOW USAGE RESOURCE GROUPS;

7. Troubleshooting the "Broker" Syntax Since StarRocks v2.4, you don't actually need a separate Broker process to load from GCS; the BEs can talk to GCS directly. However, the

WITH BROKER

syntax is still used. * Note: If you haven't defined a "broker name" in your cluster configuration, the syntax

WITH BROKER ("key"="value")

might behave differently depending on your version. Try specifying a "dummy" name if you aren't using a broker process, or ensure your storage integration is set up. * Refined Syntax Check: Ensure your GCS path is correct and accessible. For GCS, sometimes the

gs://

prefix is handled better when specific GCS properties are passed. Your properties look correct for a GCE Service Account. Summary Checklist: 1. Run

SHOW LOAD WHERE STATE = 'LOADING'

to see if the concurrency limit is hit. 2. Run

SHOW BACKENDS

to ensure BEs are healthy. 3. If the queue persists and you have resources, try increasing

max_broker_load_job_concurrency

. 4. Check the FE log (

fe.log

) for any "scheduling" or "pending" related warnings. If the job stays in

QUEUEING

for more than a few minutes with no other loads running, it usually points to the BEs being unavailable or a configuration limit being set too low. References • sql-reference/sql-statements/loading_unloading/BROKER_LOAD.md • administration/management/FE_configuration.md • loading/BrokerLoad.md • administration/management/resource_management/query_queues.md

@Rocky what is the difference between BACKENDS and compute nodes for broker load?

In StarRocks, both Backends (BEs) and Compute Nodes (CNs) can participate in a Broker Load, but they serve different roles depending on your cluster architecture. Here are the key differences: 1. Architectural Context * Backends (BE): Used in the Shared-nothing architecture. BEs are responsible for both computing (processing the load) and storage (storing the data tablets on local disks). * Compute Nodes (CN): Used primarily in the Shared-data (cloud-native) architecture. They are stateless and only handle the computation. In a shared-data cluster, CNs pull data from the source and write it to shared storage (like S3 or HDFS). _ _Note:* You can also add CNs to a Shared-nothing cluster specifically to offload computation (like queries and loads) from the BEs. 2. Data Storage vs. Processing * During Broker Load on a BE: The BE node downloads the data from the broker/cloud storage, parses it, transforms it, and then writes it to its own local disks (where the specific tablets for that table reside). * During Broker Load on a CN: The CN node downloads and processes the data but does not store it locally. * In Shared-data mode: It writes the final data to the remote shared storage. * In Shared-nothing mode (if CNs are present): The CN performs the "heavy lifting" of parsing and transforming the data, but it must then send the processed data over the network to a BE to be persisted on disk. 3. Scalability and Elasticity * BEs: Scaling BEs involves data redistribution (rebalancing tablets), which can be time-consuming. You cannot easily "burst" BEs just for a large load job. * CNs: Because they are stateless, CNs can be scaled up or down instantly. This makes them ideal for handling large, bursty Broker Load jobs. You can spin up a cluster of CNs to handle a massive data ingestion and then shut them down once the load is complete without worrying about data movement. 4. Load Job Splitting Regardless of whether you use BEs or CNs, the Frontend (FE) splits a Broker Load job into tasks based on the number of available nodes. * The formula for the number of instances is generally:

min(Amount of data / min_bytes_per_broker_scanner, Number of BE or CN nodes)

* If your cluster has both BEs and CNs, StarRocks can utilize both to parallelize the data processing. Summary Table Feature | Backend (BE) | Compute Node (CN) Primary Role | Storage + Compute | Compute Only State | Stateful (holds data) | Stateless Broker Load Action | Parses data & writes to local disk | Parses data & sends to BE or Cloud Storage Elasticity | Hard to scale quickly | Easy to scale for load bursts Architecture | Shared-nothing | Shared-data (or helper in Shared-nothing) Recommendation: If you have a high-frequency or high-volume loading requirement that fluctuates, using Compute Nodes allows you to scale your ingestion capacity independently of your storage capacity. References • sql-reference/sql-statements/loading_unloading/BROKER_LOAD.md • loading/BrokerLoad.md • sql-reference/sql-statements/cluster-management/nodes_processes/SHOW_BACKENDS.md • administration/management/monitoring/metrics.md • quick_start/shared-data.md