http://starrocks.io logo
Join Slack
Powered by
<@U0A71G31CDV> can u explain below error ```2026-...
# questions-and-troubleshooting
h
@Rocky can u explain below error
Copy code
2026-02-23T19:13:14.151899932Z W20260223 19:13:14.151205 139766133331520 stream_load_executor.cpp:118] fragment execute failed, query_id=48c40b7fc2fc44cc-aa612d384a732038, err_msg=load channel: 48c40b7f-c2fc-44cc-aa61-2d384a732038 was aborted at 1771873994, reason: transaction aborted via LakeService rpc, id=48c40b7fc2fc44cc-aa612d384a732038, job_id=5825843, txn_id: 27672336, label=ROUTINE_LOAD_API_ENDPOINT_4-5825843-48c40b7f-c2fc-44cc-aa61-2d384a732038, db=REPORTING_1
r
The error message you're seeing indicates that a Routine Load task was forcefully cancelled by the cluster's metadata management service (the LakeService) while it was being executed on a BE (Backend) node. Error Breakdown: * Component: 
stream_load_executor.cpp
— The backend logic responsible for executing the data ingestion. * Job Type: Routine Load (indicated by the label 
ROUTINE_LOAD_API_ENDPOINT_4-5825843...
). * Reason: 
transaction aborted via LakeService rpc
— This means the BE received a remote call from the FE (Frontend) or the StarOS metadata service instructing it to abort the current transaction (
txn_id: 27672336
). Possible Causes: 1. Task Timeout (Most Common): Each Routine Load task has a maximum execution time. If the task takes longer than 
routine_load_task_timeout_second
(default is usually 60 seconds), the FE will abort the transaction. If your data volume per batch is high or the network is slow, the task might expire before it finishes writing to storage. * Solution: Increase the timeout for the job:
Copy code
sql
        ALTER ROUTINE LOAD FOR your_job_name
        PROPERTIES ("routine_load_task_timeout_second" = "300");
2. Job Status Change: If someone manually executed 
PAUSE ROUTINE LOAD
or 
STOP ROUTINE LOAD
, or if the job was automatically paused due to too many error rows (exceeding 
max_error_number
), the FE will immediately abort any active transaction fragments. * Solution: Run 
SHOW ROUTINE LOAD FOR REPORTING_1.your_job_name\G
and check the 
State
and 
ReasonOfStateChanged
. 3. Conflict with Schema Change or Partition Operations: If a 
TRUNCATE TABLE
, 
DROP PARTITION
, or 
ALTER TABLE
(Schema Change) operation was triggered on the table 
REPORTING_1
while the load was running, the transaction might be aborted to maintain consistency. 4. FE Leader Switch: If the FE Leader restarted or a new Leader was elected at 
19:13:14
, it might have aborted all in-flight transactions that it couldn't verify. 5. Metadata Service (StarOS) Issues: In StarRocks' shared-data (Cloud-Native) architecture, the LakeService manages data consistency on object storage. If the BE loses connection to the FE/LakeService or if there is a conflict in the metadata version, the service will abort the transaction. Recommended Troubleshooting Steps: 1. Check Job Status:
Copy code
sql
    SHOW ROUTINE LOAD FROM REPORTING_1 WHERE label LIKE '%5825843%';
Look at 
ErrorLogUrls
if the state is 
PAUSED
. 2. Inspect FE Logs: Search your FE leader's 
fe.log
or 
fe.warn.log
around 
2026-02-23 19:13:14
for the Transaction ID 
27672336
. This will reveal the exact reason why the FE decided to abort the transaction (e.g., "transaction timeout", "label already exists", or "tablet is not healthy"). 3. Check Resources: Ensure the BE has enough memory and CPU. If the BE is under extreme load, it might delay RPC responses, leading the FE to believe the task has hung and aborting it. References • faq/resource_isolation_faq.md • administration/Meta_recovery.md • administration/management/BE_configuration.md • knowledge/trouble_shooting/fe_deadlock_problems.md
h
are you sure it is due to timeout ? how can i validate this logic @Rocky
r
To validate if an error is specifically caused by a timeout in StarRocks, you can follow these steps to correlate the error symptoms with the system configuration. 1. Identify the Error Message The first sign of a timeout is the specific error message returned to the client or found in the logs. * Query Timeout: 
ERROR 1064 (HY000): Query exceeded time limit of 300 seconds
(The value 300 depends on your configuration). * Planner Timeout: 
StarRocks planner use long time 3000 ms in logical phase
or 
The query exceeds the maximum time limit for query planning
. * Load Timeout: For Stream Load or Broker Load, you might see 
Status: CANCELLED
with a reason like 
timeout
. * Insert Timeout: 
Insert into statement timeout...
. 2. Check the Current Timeout Configuration You can validate the "logic" by checking what the current limits are for your session or the global system. Compare these values to how long the task was running before it failed. Run the following SQL to see relevant timeout variables:
Copy code
sql
SHOW VARIABLES LIKE '%timeout%';
Key variables to look for: * 
query_timeout
: (Default 300s) The maximum time a query can run. * 
insert_timeout
: (Default 3600s) Specifically for 
INSERT INTO ... SELECT
or 
INSERT INTO ... VALUES
statements. * 
new_planner_optimize_timeout
: (Default 3000ms) The limit for the SQL optimizer to generate a plan. * 
wait_timeout
: The idle time before a connection is closed. 3. Compare with Actual Execution Time To confirm it is a timeout, check how long the query actually ran. * Audit Log: Check 
fe/log/fe.audit.log
. Look for the 
QueryTime
(ms) column for the specific 
query_id
. If 
QueryTime
is equal to or slightly higher than your 
query_timeout
, it is definitely a timeout. * Profile List: Run 
SHOW PROFILELIST;
to see the duration of recent queries. If a query has a state of 
Error
and the duration matches your timeout setting, the logic is confirmed. * Detailed Profile: If the query is still running or recently failed, you can inspect the profile:
Copy code
sql
    SET enable_profile = true;
    -- After the query fails/runs
    SELECT get_query_profile('<query_id>');
4. Search FE/BE Logs for "Cancel" or "Timeout" If you suspect a system-level or communication timeout, check the logs on the FE (Frontend) and BE (Backend) nodes: * FE Log (
fe.log
or 
fe.warn.log
): Look for 
Cancel query
, 
query timeout
, or 
state changed from RUNNING to CANCELLED
. * BE Log (
<http://be.INFO|be.INFO>
or 
be.WARNING
): Look for fragments being cancelled due to timeout. Search for the phrase: 
query timeout
or 
reached timeout
. 5. Validation Test (The "Smoking Gun") To verify the logic, try to temporarily increase the timeout for your session and rerun the task:
Copy code
sql
-- Increase query timeout to 1 hour for the current session
SET query_timeout = 3600;

-- If it's a planner issue
SET new_planner_optimize_timeout = 10000;

-- Rerun your query/load
If the query succeeds after increasing the value, you have validated that the previous failure was strictly due to the timeout limit. Summary Table for Validation Symptom | Variable to Check | Log to Validate Query fails at exactly X seconds | 
query_timeout
| 
fe.audit.log
(QueryTime) SQL fails during "Planning" | 
new_planner_optimize_timeout
| 
fe.log
INSERT INTO
fails after a long time | 
insert_timeout
| 
fe.audit.log
Routine/Stream Load fails | 
timeout
(in job properties) | 
SHOW LOAD
or 
information_schema.loads
References • best_practices/query_tuning/query_profile_overview.md • faq/Sql_faq.md • best_practices/query_tuning/query_profile_text_based_analysis.md
h
routine load was showing this error for above log
Copy code
"OtherMsg": "[2026-02-23 19:12:44] [task id: 48c40b7f-c2fc-44cc-aa61-2d384a732038] [txn id: -1] previous task aborted because of kafka consume failed, err: FindCoordinator response error: Local: Bad message format",
@Rocky
r
Hi! I'm Rocky. The error 
FindCoordinator response error: Local: Bad message format
in StarRocks Routine Load usually indicates a network or resolution issue between the StarRocks nodes (specifically the BEs) and the Kafka brokers. Cause This error often occurs because Kafka brokers typically communicate using hostnames. When a StarRocks BE attempts to connect, it receives metadata from Kafka (the "FindCoordinator" step) that contains the broker's hostname. If the StarRocks node cannot resolve that hostname to an IP address, the underlying Kafka client (
librdkafka
) returns the "Bad message format" error. Solution You should ensure that all StarRocks nodes (FE and BE) can resolve the hostnames of all Kafka brokers. 1. Check Hostname Resolution: Log into your StarRocks BE nodes and try to 
ping
or 
telnet
the Kafka broker hostnames. 2. Update `/etc/hosts`: If you are not using a DNS server that includes your Kafka brokers, manually add the mapping of Kafka broker hostnames to their IP addresses in the 
/etc/hosts
file on all StarRocks FE and BE nodes.
Copy code
bash
    # Example /etc/hosts entry
    192.168.1.10 kafka-broker-1
    192.168.1.11 kafka-broker-2
3. Check Kafka Advertised Listeners: Ensure that the Kafka brokers are configured with 
advertised.listeners
that are reachable by the StarRocks cluster. If Kafka is advertising a hostname that the StarRocks BEs cannot reach, you will encounter this error. If the issue persists after fixing hostname resolution, verify that there isn't a version mismatch or security (SSL/SASL) configuration error, though "Bad message format" is most commonly tied to the resolution issue mentioned above. References • faq/loading/Routine_load_faq.md • quick_start/routine-load.md
h
dns is fine i am able to find @Rocky
r
That's great to hear! Resolving DNS issues is often the trickiest part of setting up the network for a StarRocks cluster. If you are configuring your cluster using hostnames or FQDNs, here are a couple of tips to keep in mind: * FQDN Access: If you want StarRocks to use hostnames instead of IP addresses for internal communication, ensure you have set 
enable_fqdn_mode = true
in your FE and BE configurations. * DNS Caching: If you are in a dynamic environment like Kubernetes, StarRocks (starting in v3.1.11+/v3.2.4+) has a parameter 
dns_cache_ttl_seconds
in the FE configuration to control how long successful DNS lookups are cached. Are you currently in the middle of adding nodes to the cluster, or is there a specific connectivity check you'd like to run next? I'm here to help with any query or configuration questions! References • administration/management/FE_configuration.md • deployment/environment_configurations.md • administration/management/enable_fqdn.md
h
fine mean it was correct earlier . what can be other issues ?
@Rocky
r
Hello, @Harit Rajkotiya! How can I assist you today?
h
fine mean it was correct earlier . what can be other issues ? @Rocky
r
If the previous configuration or syntax was correct, there are several other common areas in StarRocks where issues typically arise. Depending on what you are seeing (errors, slowness, or data issues), here are the most likely "other" culprits: 1. Resource & Memory Limits If your query is failing with a specific error message, it’s often related to memory: * 
exec_mem_limit
: By default, StarRocks limits the memory a single query can use. If you see "Mem usage has exceed the limit of single query," you may need to increase this session variable (e.g., 
SET exec_mem_limit = 8589934592;
). * BE Total Memory: If the Backend (BE) itself is low on memory, it might reject queries or trigger a process kill. You can check BE memory usage via the 
metrics
endpoint or the 
mem_tracker
. * Query Pool Limit: If you use Resource Groups, a query might be queued or rejected if the pool's limit is hit. 2. Metadata and Frontend (FE) Issues Sometimes the problem isn't the query itself, but the FE's ability to plan it: * FE JVM GC: If the FE is experiencing "Full GC," query planning can become extremely slow or time out (look for "planner use long time" in logs). Increasing the FE JVM heap size in 
fe.conf
(e.g., to 16GB) often fixes this. * Catalog Sync: If you are querying external tables (like Hive or Iceberg), there might be a metadata synchronization delay or a credential issue. 3. Data Distribution (Skew) If a query is "correct" but extremely slow: * Data Skew: If your 
DISTRIBUTED BY
key has many duplicate values, one BE might be doing 90% of the work while others stay idle. Check 
SHOW TABLE STATUS
and the query profile. * Bucket Count: Having too many small tablets (over-sharding) or too few large tablets can significantly hurt performance. * Partition Pruning: Ensure your 
WHERE
clause uses the partition column so StarRocks doesn't scan the entire table. 4. Materialized View (MV) Logic If you are using Asynchronous MVs: * Refresh Failures: The MV might be "Inactive" if a schema change occurred on the base table. Use 
ALTER MATERIALIZED VIEW <mv_name> ACTIVE;
. * Rewrite Failure: If a query isn't being rewritten to use an MV, it might be due to 
query_rewrite_consistency
settings or the query containing unsupported functions (like window functions or nested aggregations). 5. Loading & Locking * Transaction Conflicts: If you are running heavy 
DELETE
or 
UPDATE
operations while querying, you might encounter lock contention. * Compaction Backlog: If data is being loaded very rapidly, the BE might fall behind on "compaction," leading to a high number of versions and slow read performance. Check 
show proc "/be_nodes";
to see if 
TabletMaxVersion
is very high. How to narrow it down: To give you a more specific answer, could you provide: 1. The specific error message (if any). 2. The Query Profile (by running 
SET enable_profile = true;
then 
ANALYZE PROFILE FROM 'query_id';
). 3. The Table Schema (
SHOW CREATE TABLE ...
). References • loading/loading_introduction/troubleshooting_loading.md • best_practices/query_tuning/query_plan_intro.md
h
"OtherMsg": "[2026-02-23 191244] [task id: 48c40b7f-c2fc-44cc-aa61-2d384a732038] [txn id: -1] previous task aborted because of kafka consume failed, err: FindCoordinator response error: Local: Bad message format", error is this so i grep the logs on the basis of id above 2026-02-23T191314.151899932Z W20260223 191314.151205 139766133331520 stream_load_executor.cpp:118] fragment execute failed, query_id=48c40b7fc2fc44cc-aa612d384a732038, err_msg=load channel: 48c40b7f-c2fc-44cc-aa61-2d384a732038 was aborted at 1771873994, reason: transaction aborted via LakeService rpc, id=48c40b7fc2fc44cc-aa612d384a732038, job_id=5825843, txn_id: 27672336, label=ROUTINE_LOAD_API_ENDPOINT_4-5825843-48c40b7f-c2fc-44cc-aa61-2d384a732038, db=REPORTING_1 i got this can u explain what is happenign @Rocky
r
Based on the error logs you've provided, here is a breakdown of what is happening: 1. Root Cause: 
FindCoordinator response error: Local: Bad message format
 This is the primary error. It originates from the Kafka client (librdkafka) used by StarRocks. When a Routine Load task starts, it needs to find the "Coordinator" (the Kafka broker responsible for managing the consumer group). The "Bad message format" error typically occurs for one of the following reasons: * Hostname Resolution Issues (Most Common): Kafka brokers often return their hostnames to the client. If the StarRocks BE nodes cannot resolve these Kafka hostnames to IP addresses, the communication fails with this cryptic message. * Protocol Mismatch: There might be a mismatch between the Kafka version and the configuration, or a security protocol issue (e.g., trying to connect via PLAINTEXT to an SSL-enabled port, or vice versa). * SASL/Security configuration: Incorrect SASL credentials or mechanism can sometimes lead to the broker returning an unexpected response that the client interprets as a bad format. 2. Secondary Error: 
transaction aborted via LakeService rpc
 This error in your BE logs is a consequence, not the cause. Because the Kafka consumption failed (due to the "Bad message format" error above), the StarRocks Routine Load scheduler aborted the transaction. Since you are likely using a StarRocks version with Shared-data architecture (or StarRocks 3.x Lake nodes), it uses the 
LakeService
to manage transactions. ────────── How to Fix This Step 1: Verify Hostname Resolution The most frequent fix is ensuring every StarRocks node (FE and BE) can resolve the hostnames of your Kafka brokers. 1. Get the list of Kafka broker hostnames. 2. On every StarRocks BE and FE node, try to ping those hostnames. 3. If they don't resolve, add them to the 
/etc/hosts
file on all StarRocks nodes:
Copy code
bash
   # Example /etc/hosts entry
   192.168.1.10 kafka-broker-1
   192.168.1.11 kafka-broker-2
Step 2: Check Kafka Security Protocol If you are using SSL or SASL, ensure your 
CREATE ROUTINE LOAD
statement includes the correct properties. A common missing property for SSL is:
Copy code
sql
"property.ssl.endpoint.identification.algorithm" = "none"
Also, ensure 
property.security.protocol
matches your Kafka cluster's requirements (e.g., 
SASL_PLAINTEXT
or 
SASL_SSL
). Step 3: Check Kafka Version Compatibility If your Kafka cluster is very old (pre-0.10), you may need to specify the broker version in the Routine Load properties so the client knows which protocol features to use:
Copy code
sql
"property.broker.version.fallback" = "0.10.0.0"
Next Steps for Troubleshooting To see exactly why the connection is failing, you can check the status of the Routine Load job:
Copy code
sql
SHOW ROUTINE LOAD FOR your_job_name\G
Look at the 
ReasonOfMultiHeaderErrors
or 
ErrorLogUrls
fields for more specific details from the FE's perspective. References • faq/loading/Routine_load_faq.md • administration/Meta_recovery.md • administration/management/FE_configuration.md
🥹 1
Open in Slack
PreviousNext
Powered by