http://starrocks.io logo
Join Slack
Powered by
<@U0A71G31CDV> I was trying to create the backup r...
# questions-and-troubleshooting
h
@Rocky I was trying to create the backup repo linked to s3 bucket and got this error. I have checked the networking connectivity and bucket permissions and they are all good.
Copy code
ERROR 5064 (42000): Failed to create repository: write exception: could not open file <s3a://bucket/_starrocks_repository/_repo_info>
r
Hi there! I'm Rocky. This error usually occurs when the StarRocks Broker (or the internal S3 client) is unable to initialize the repository by writing a metadata file (
_repo_info
) to your bucket. Since you've already verified networking and basic permissions, here are the most common reasons this specific error persists: 1. Missing Broker Name In the 
CREATE REPOSITORY
statement, you must specify the name of a registered broker. Even if you are using built-in S3 support for other features, the Backup/Restore process typically requires a Broker component. * Check registered brokers: Run 
SHOW BROKER;
to see your broker names. * Correct Syntax:
Copy code
sql
    CREATE REPOSITORY my_s3_repo
    WITH BROKER "your_broker_name" -- Make sure this matches SHOW BROKER
    ON LOCATION "<s3a://bucket/_starrocks_repository/>"
    PROPERTIES (
        "aws.s3.access_key" = "your_ak",
        "aws.s3.secret_key" = "your_sk",
        "aws.s3.region" = "us-east-1"
    );
2. Region Mismatch The 
aws.s3.region
property is mandatory and must exactly match the region where your bucket resides (e.g., 
us-west-2
). A mismatch often results in a "could not open file" error because the S3A client attempts to connect to the wrong endpoint. 3. S3A Endpoint Configuration If you are using a VPC Endpoint for S3 or an S3-compatible storage, you may need to explicitly provide the endpoint: * Add 
"aws.s3.endpoint" = "<http://s3.us-east-1.amazonaws.com|s3.us-east-1.amazonaws.com>"
to your 
PROPERTIES
. * Note: For AWS S3, do not include 
https://
in the endpoint string. 4. Permission Granularity StarRocks needs to be able to create and delete files to manage the repository. Ensure your IAM policy includes these actions for the specific path: * 
s3:PutObject
* 
s3:GetObject
* 
s3:ListBucket
* 
s3:DeleteObject
5. Using 
s3a://
vs 
s3://
 You are already using 
s3a://
, which is correct for AWS S3 and GCS repositories in StarRocks. If you happen to be using MinIO, try switching the prefix to 
s3://
as some versions handle it differently for MinIO. Quick Tip: Check the Broker's log file (
broker.log
or 
apache_hdfs_broker.log
) on your worker nodes. It often contains a much more descriptive Java stack trace (like a 403 Forbidden or 404 Not Found) that explains exactly why the file "could not be opened." Let me know if you're able to see more details in the broker logs! References • sql-reference/sql-statements/backup_restore/CREATE_REPOSITORY.md • administration/management/Backup_and_restore.md • administration/management/BE_configuration.md • faq/Deploy_faq.md • administration/management/FE_configuration.md • faq/Exporting_faq.md
h
@Rocky I was trying to create a backup to a repo linked to s3 and got this error. I have checked the networking connectivity and bucket permissions and they are all good.
Copy code
Status: [COMMON_ERROR, msg: fail to open writer for <s3a://bucket/__starrocks_repository/snapshot/__meta.part>]
r
The error 
fail to open writer for <s3a://...>
typically occurs when the StarRocks Broker (or the FE/BE if running Broker-less) cannot initialize the write stream to your S3 bucket. While you've checked permissions and connectivity, this error often stems from specific configuration nuances in the 
CREATE REPOSITORY
statement or the underlying S3A protocol. Here are the most common causes and troubleshooting steps: 1. Missing or Incorrect 
aws.s3.region
 For AWS S3, specifying the region is often mandatory, even if your network is internal. If the region is missing or incorrect, the S3A client may fail to sign requests properly, resulting in a "fail to open writer" error. Solution: Ensure your repository includes the 
aws.s3.region
property:
Copy code
sql
CREATE REPOSITORY `your_repo_name`
WITH BROKER
ON LOCATION "<s3a://bucket/path/>"
PROPERTIES(
    "aws.s3.access_key" = "your_access_key",
    "aws.s3.secret_key" = "your_secret_key",
    "aws.s3.region" = "us-east-1" -- Ensure this matches your bucket's region
);
2. S3A Endpoint Configuration If you are using a non-standard S3-compatible storage (like MinIO) or a VPC endpoint, you must provide the 
aws.s3.endpoint
. * For AWS S3: Generally, you don't need the endpoint if the region is correct, but for some private links, you might. * For MinIO/Others: Use 
aws.s3.endpoint
= 
http://<ip>:<port>
. 3. Broker Service Logs Since the error is prefixed with 
COMMON_ERROR
, the high-level message is often generic. The actual underlying exception (e.g., 
403 Forbidden
, 
400 Bad Request
, or 
Clock Skew
) is logged in the Broker's log files. * Check 
apache_hdfs_broker.out
or 
broker.log
on the nodes where your Broker is running. * Look for the specific timestamp of the failed 
BACKUP
command to find the Java stack trace. 4. Property Naming Convention Depending on your StarRocks version and whether you are using a Broker, the S3 properties might expect different prefixes. If 
aws.s3.access_key
doesn't work, try the Hadoop-style 
fs.s3a
prefix:
Copy code
sql
PROPERTIES(
    "fs.s3a.access.key" = "your_access_key",
    "fs.s3a.secret.key" = "your_secret_key",
    "fs.s3a.endpoint" = "<http://s3.us-east-1.amazonaws.com|s3.us-east-1.amazonaws.com>"
);
5. IAM Roles and Permissions If you are using IAM Instance Profiles (instead of keys), ensure: 1. 
"aws.s3.use_instance_profile" = "true"
is set in properties. 2. The IAM Role is attached to the Broker nodes (not just the FE/BE nodes). 3. The role has 
s3:PutObject
, 
s3:GetObject
, 
s3:ListBucket
, and 
s3:DeleteObject
permissions on the bucket and the specific path. 6. Protocol Prefix StarRocks requires the 
s3a://
prefix for AWS S3 repositories. Ensure you are not using 
s3://
in the 
ON LOCATION
clause, as the internal S3A client specifically looks for the 
s3a
schema. Quick Check: Try to run a simple 
SHOW SNAPSHOT ON your_repo_name;
or 
CREATE REPOSITORY
with a new empty path to see if the initialization succeeds. If it fails immediately, it is almost always a credential/region/endpoint mismatch in the 
PROPERTIES
block. References • faq/Exporting_faq.md • loading/s3.md • faq/operation_maintenance_faq.md • faq/Deploy_faq.md • administration/Meta_recovery.md
h
@Rocky Why starrocks generates many large fe.internal.log
r
The 
fe.internal.log
file in StarRocks is specifically designed to record internal FE operations that are separate from user-initiated queries. If you are seeing many large 
fe.internal.log
files, it is usually due to one of the following reasons: 1. High Frequency of Statistics Collection The most common reason for a large 
fe.internal.log
is the Statistics module (
internal.statistic
). StarRocks automatically collects statistics (like row counts and column distributions) to help the CBO (Cost-Based Optimizer) generate efficient query plans. * If your cluster has a very large number of tables, partitions, or frequent data ingestion, the internal SQLs used to collect these statistics will generate a high volume of logs. 2. Internal Audit Logging The 
internal.base
module logs internal system-level SQL executions. If the system is performing many background tasks (such as metadata synchronization or routine maintenance), these logs can grow quickly. 3. Retention Settings By default, StarRocks keeps a significant amount of these logs: * 
internal_log_roll_num
: Defaults to 90. This means StarRocks will keep up to 90 rolled-over log files. * 
internal_log_delete_age
: Defaults to 7 days. Logs older than this are deleted. * 
sys_log_roll_interval
/ 
log_roll_size_mb
: Logs typically roll daily or when they reach ~1GB. ────────── How to Manage or Reduce the Size You can adjust these settings in your 
fe.conf
file (requires an FE restart): 1. Reduce the number of retained files: Decrease 
internal_log_roll_num
to a smaller value (e.g., 10 or 20) to save disk space.
Copy code
properties
    internal_log_roll_num = 10
2. Shorten the retention period: Reduce 
internal_log_delete_age
to delete logs sooner.
Copy code
properties
    internal_log_delete_age = 3d
3. Disable specific internal modules (Not recommended for troubleshooting): If you find that the statistics logs are unnecessary for your current monitoring, you can modify 
internal_log_modules
. The default is 
{"base", "statistic"}
. Removing 
statistic
would stop logging stats collection, but this makes it harder to debug optimizer issues.
Copy code
properties
    internal_log_modules = {"base"}
4. Enable Compression: Ensure log compression is enabled to reduce the footprint of older logs on the disk.
Copy code
properties
    sys_log_enable_compress = true
Summary: Large 
fe.internal.log
files are generally a sign of a very active cluster (lots of metadata or statistics updates). Adjusting the retention count (
internal_log_roll_num
) is the safest way to limit disk usage without losing recent diagnostic information. References • administration/management/logs.md • administration/management/FE_configuration.md
h
@Rocky Why starrocks backp is stuck on UPLOADING
r
When a StarRocks BACKUP job is stuck in the UPLOADING state, it means the Backend (BE) nodes are currently transferring snapshot files of the tablets to the remote storage repository (S3, OSS, or a Broker-mediated file system). Here are the steps to diagnose and resolve the issue: 1. Check the Job Details Run the following SQL command to see the current status of the backup job:
Copy code
sql
SHOW BACKUP FROM <your_database_name>;
Pay close attention to these columns: * Progress: Check if the percentage is increasing. If it's moving slowly, the backup is working but may be bottlenecked by data size or network. * UnfinishedTasks: This lists the IDs of the sub-tasks that are still running. * TaskErrMsg: If some sub-tasks have failed, the error message will appear here. 2. Identify the Problematic BE If 
UnfinishedTasks
shows specific IDs, StarRocks is waiting for those tasks to complete. You can identify which BEs are handling these tasks by looking at the FE logs (
fe.log
). Search for the 
JobId
or the specific task IDs to find which BE IP is responsible for the upload. 3. Check BE and Broker Logs Once you identify the BE node: * BE Logs: Check 
<http://be.INFO|be.INFO>
or 
be.WARN
on that node. Search for "upload" or "snapshot" to see if there are timeout errors, permission denied errors, or connection resets. * Broker Logs: If you are using a Broker to connect to HDFS or storage, check the Broker's logs. If the Broker is down or misconfigured, the upload will hang. 4. Common Causes * Large Number of Tablets: If your table has many partitions or tablets, StarRocks creates a snapshot for each. Thousands of small files can significantly slow down the 
UPLOADING
phase. * Network Bandwidth: High network latency or restricted bandwidth between BEs and the storage repository. * Storage Permissions/Quota: Ensure the credentials used in 
CREATE REPOSITORY
still have write permissions and that the storage bucket hasn't hit its quota or rate limit. * Disk Space on BE: BEs need enough local disk space to create the snapshot before uploading. 5. Performance Tuning If backups are consistently slow in the 
UPLOADING
phase, you can adjust the following parameters: * 
upload_worker_count
(BE configuration): In 
be.conf
, this defines the number of threads for upload tasks. The default is 
0
(equivalent to the number of CPU cores). You can try increasing this to speed up parallel uploads. * 
max_upload_task_per_be
(FE configuration): This is an FE dynamic parameter that limits how many upload tasks are assigned to a BE at once.
Copy code
sql
    ADMIN SET FRONTEND CONFIG ("max_upload_task_per_be" = "10"); -- Example
6. How to Cancel If the job is truly stuck and you want to restart it, use:
Copy code
sql
CANCEL BACKUP FROM <your_database_name>;
Rocky's Tip: Always check the connectivity from the BE nodes to your storage endpoint (e.g., using 
curl
or 
ping
) to rule out basic network blocks or firewall issues! References • administration/management/BE_configuration.md • administration/management/FE_configuration.md • sql-reference/sql-statements/backup_restore/BACKUP.md • administration/management/monitoring/metrics.md • sql-reference/sql-statements/backup_restore/SHOW_BACKUP.md • administration/management/Backup_and_restore.md • unloading/Export.md
h
@Rocky This message came from BE. Is this something concerning?
Copy code
automatic partition rpc end response
r
This log message, 
automatic partition rpc end response
, is not concerning by itself. It is an informational message related to StarRocks' Expression Partitioning (also known as Automatic Partitioning) feature. What it means When you use Expression Partitioning (e.g., 
PARTITION BY dt
), StarRocks automatically creates new partitions as data is loaded. The process works like this: 1. BE Detection: During a data load (like Stream Load or Broker Load), the Back-End (BE) node detects a row with a partition value that doesn't exist yet. 2. RPC Request: The BE sends an RPC (Remote Procedure Call) to the Front-End (FE) to request the creation of that new partition. 3. Log Message: The log 
automatic partition rpc end response
simply indicates that the FE has finished processing the request and the BE has received the response confirming the partition was handled (either created or already exists). When to ignore it * If your load jobs are completing successfully. * If you are loading data for new time periods (e.g., a new day or hour) where partitions need to be created on the fly. * If there are no accompanying 
ERROR
or 
WARNING
logs. When to investigate While the message itself is normal, you should look closer if: * Load Latency: If you see thousands of these messages in a short burst and your load is slow, you might be creating too many partitions at once. You can check the FE parameter 
auto_partition_max_creation_number_per_load
(default 4096). * Accompanying Errors: If the log is followed by a "timeout" or "failed to create partition" error, it indicates the FE is struggling to keep up with metadata updates. * Cardinality Issues: If you accidentally partitioned by a high-cardinality column (like a 
user_id
or 
timestamp
with seconds), this log will appear constantly as the system struggles to create an overwhelming number of partitions. Summary: This is a routine trace of the automated partitioning handshake between the BE and FE. Unless your loads are failing, you can safely ignore it. References • administration/management/BE_configuration.md
h
@Rocky Is this message from BE concerning?
Copy code
cancel LocalTabletsChannel reason: empty
r
The message 
cancel LocalTabletsChannel reason: empty
in the StarRocks Backend (BE) logs is generally not a cause for concern. It is an informational message related to the data loading process (such as Stream Load, Broker Load, or Routine Load). Here is a breakdown of what it means and why it appears: 1. What it means A 
LocalTabletsChannel
is an internal structure the BE uses to manage data being written to its local tablets during a load job. The message indicates that this specific channel was closed or cancelled because no data was ever received for the tablets it was supposed to handle. 2. Common Reasons * Empty Source Data: The load job was initiated, but the source file or stream contained no rows. * Data Distribution/Filtering: Even if the overall load job has data, StarRocks distributes data across different BEs based on your partitioning and bucketing. If a specific BE doesn't receive any rows that belong to its local tablets (due to the way data is hashed or filtered), it will eventually close its empty channel. * Job Cancellation: If a load job is cancelled (either manually by a user or because of an error on another node), the FE will notify all BEs to cancel their respective channels. If a BE hadn't received data yet, it logs "reason: empty". 3. When should you care? * If your load jobs are successful: You can safely ignore this log. It is simply the BE cleaning up resources for a task that had nothing to do. * If your load jobs are failing: * Check for a "Reason" in the 
SHOW LOAD
output or the Stream Load response. * If you see an error like 
all partitions have no load data
, it confirms the source was empty or all rows were filtered out. * If you want empty loads to be treated as successful instead of errors, you can check the FE configuration 
empty_load_as_error
(which is 
true
by default). Summary: Unless you are seeing corresponding load failures or "all partitions have no load data" errors that you don't expect, this log is just a routine part of how StarRocks manages its internal loading channels. References • faq/loading/Loading_faq.md • administration/management/BE_configuration.md • administration/management/FE_configuration.md
h
@Rocky Is this message from BE log concerning?
Copy code
bvar is busy at sampling for 2 seconds!
r
The log message 
bvar is busy at sampling for 2 seconds!
is a warning from the bRPC library (used by StarRocks for internal communication) and typically indicates that the background thread responsible for collecting performance metrics (
bvars
) is experiencing delays. Is it concerning? In most cases, no, it is not a fatal error, and it does not affect data correctness. However, it is a "canary in the coal mine" for performance: * System Pressure: It usually occurs when the Backend (BE) is under extreme CPU pressure or resource contention. If the CPU is pegged at 100%, the sampling thread may not get enough cycles to complete its work on time. * Metric Overload: It can also happen if there are an unusually high number of active metrics or RPC connections being tracked simultaneously. What should you check? If you see this message frequently, you should investigate the health of that specific BE: 1. CPU Usage: Check if the BE node is hitting high CPU utilization. Frequent occurrences often correlate with CPU saturation. 2. Query Concurrency: Check if there are a massive number of concurrent queries or data loading tasks running. 3. RPC Latency: Use the following SQL to see if there is high latency in bRPC operations:
Copy code
sql
    SELECT * FROM information_schema.be_bvars WHERE NAME LIKE '%rpc_latency%';
4. Other Logs: Look for related messages like 
The server is overcrowded
or 
brpc failed
, which might confirm that the node is struggling to keep up with the load. Summary If this happens only occasionally during peak loads, you can generally ignore it. If it happens constantly and is accompanied by query slowdowns, it's a sign that you may need to scale your cluster, optimize your queries, or investigate potential resource bottlenecks (like I/O or memory pressure) that are indirectly slowing down the CPU threads. References • administration/management/monitoring/metrics.md • faq/Sql_faq.md • knowledge/usage/best-practice.md • sql-reference/information_schema/be_bvars.md • administration/management/FE_configuration.md • faq/Deploy_faq.md • administration/management/BE_configuration.md
h
@Rocky What does this error mean from FE?
Copy code
[ConnectProcessor.computeStatementDigest():308] Failed to compute statement digest
 java.lang.NullPointerException
r
This error message from the StarRocks Front End (FE) is related to the SQL Digest feature, which was introduced in version 3.3.6. What does it mean? The SQL Digest feature is designed to generate a unique fingerprint (digest) for SQL statements by removing literal parameters. This allows StarRocks to group similar queries for auditing and performance analysis (e.g., tracking the most time-consuming SQL patterns). The error 
[ConnectProcessor.computeStatementDigest():308] Failed to compute statement digest java.lang.NullPointerException
means that while the FE was trying to "normalize" a SQL statement to create this fingerprint, it encountered an unexpected 
NullPointerException
(NPE) in the internal code. Possible Causes 1. Complex or Edge-case SQL Syntax: A specific SQL query pattern (such as certain complex 
INSERT
statements, specific 
WITH
clauses, or unusual nested expressions) may be triggering a code path that isn't handled correctly by the digest generator. 2. Software Bug: Since this feature is relatively new (available from v3.3.6+), it is likely a bug in the specific version of StarRocks you are running. Recommended Actions 1. Verify Impact: Usually, this error is non-fatal for the query execution itself. The query should still run, but the 
Digest
field in your 
fe.audit.log
will be empty or failed for that specific statement. 2. Workaround (Disable the Feature): If these errors are cluttering your logs and you do not rely on SQL Digests for auditing, you can disable the feature dynamically:
Copy code
sql
    ADMIN SET FRONTEND CONFIG ('enable_sql_digest' = 'false');
To make this permanent, add 
enable_sql_digest = false
to your 
fe.conf
and restart the FE. 3. Identify the Query: Check your 
fe.audit.log
around the same timestamp as the error to identify the specific SQL statement being executed. 4. Upgrade: Check the release notes for later patch versions of your current major branch (e.g., latest 3.3.x or 3.5.x) to see if a fix for 
computeStatementDigest
has been included. If you can identify the specific SQL that causes this, it is highly recommended to report it as an issue to the StarRocks community. References • administration/sql_digest.md • sql-reference/sql-functions/crytographic-functions/sha2.md • sql-reference/sql-functions/crytographic-functions/sm3.md • sql-reference/sql-functions/crytographic-functions/md5sum_numeric.md • sql-reference/sql-functions/crytographic-functions/md5.md • sql-reference/sql-functions/crytographic-functions/md5sum.md • sql-reference/sql-functions/string-functions/crc32.md • sql-reference/sql-functions/scalar-functions/hll_empty.md
h
@Rocky What do these mean from FE?
Copy code
2026-02-24 14:22:31.596Z WARN (thrift-server-pool-63|387) [LeaderImpl.finishTask():237] cannot find task. type: PUBLISH_VERSION, backendId: 12345, signature: 5343206
 2026-02-24 14:22:37.830Z WARN (ReportHandler|62) [ReportHandler.deleteFromMeta():1175] report Version from be: 12345 is outdated, report version in request: 17704118734039, latest report version: 17704118734045, ignore tablet: 12327658
r
Hello! I'm Rocky, and I can help you decode those StarRocks FE logs. These are common warning logs that generally relate to the communication and state synchronization between the FE (Front End) and BE (Backend) nodes. Here’s what each of those logs means: 1. 
cannot find task. type: PUBLISH_VERSION
Copy code
text
2026-02-24 14:22:31.596Z WARN (thrift-server-pool-63|387) [LeaderImpl.finishTask():237] cannot find task. type: PUBLISH_VERSION, backendId: 12345, signature: 5343206
* What it means: When a transaction is committed, the FE sends a 
PUBLISH_VERSION
task to the BEs to make the data visible. This warning indicates that a BE (ID 12345) has responded that it finished the task, but the FE no longer has a record of this specific task (identified by the 
signature
). * Why it happens: This usually occurs due to a timeout. If the BE takes longer than the expected threshold to finish the publish task, the FE might have already removed the task from its internal "pending" list or marked the transaction as failed/timed out. When the BE finally finishes and sends the acknowledgment, the FE simply reports that it can't find the original task entry. * Action: If this happens occasionally, it is usually harmless. However, if you see it frequently alongside slow loading or "publish version timeout" errors, you may need to check the BE disk I/O or increase the 
publish_version_timeout_second
configuration in the FE. 2. 
report Version from be is outdated
Copy code
text
2026-02-24 14:22:37.830Z WARN (ReportHandler|62) [ReportHandler.deleteFromMeta():1175] report Version from be: 12345 is outdated, report version in request: 17704118734039, latest report version: 17704118734045, ignore tablet: 12327658
* What it means: BE nodes periodically send "reports" to the FE containing their current status (e.g., tablet information, disk usage). Each report has an incremental version number. This log means the FE received a report from BE 12345 that has a lower version number (
...4039
) than a report it has already processed (
...4045
). * Why it happens: This is typically caused by network jitter or latency. An older report packet was delayed in the network and arrived at the FE after a newer report packet had already been received and processed. * Action: The FE correctly ignores the stale report to prevent overwriting its metadata with outdated information. This is a self-healing mechanism and is generally safe to ignore unless you are seeing persistent network instability between your FE and BE nodes. In summary, both of these logs are warnings about asynchronous events arriving out of their expected window and do not necessarily indicate a critical failure in the cluster. References • administration/management/BE_configuration.md • deployment/upgrade.md • deployment/downgrade.md
h
@Rocky What does this error mean from FE
Copy code
2026-02-24 13:30:45.952Z ERROR (JournalWriter|123) [JournalWriter.abortJournalTask():196] failed to commit journal after retried 3 times!
r
The error message 
failed to commit journal after retried 3 times!
is a critical error from the StarRocks Frontend (FE). It indicates that the Leader FE failed to write its metadata changes (edit logs) to the backend storage (typically BDBJE) even after multiple retry attempts. What it means In StarRocks, the Leader FE is responsible for all metadata modifications (e.g., creating tables, starting load jobs, schema changes). These changes are written to a "journal." If the Leader cannot commit these changes to the journal, it cannot guarantee metadata consistency or persistence. Consequently, the FE will usually stop providing services or exit to prevent further issues. Common Causes 1. Disk Issues (Most Common): * Disk Full: The disk partition where 
meta_dir
(specified in 
fe.conf
) is located has run out of space. * Slow I/O: The disk is experiencing extreme latency or is stuck, preventing the BDBJE transaction from committing within a reasonable time. * Permission Issues: The FE process no longer has write permissions to the metadata directory. 2. Quorum/Network Loss: * In a High Availability (HA) setup with multiple FEs, the Leader must synchronize the journal to a majority (quorum) of Follower nodes. If the Leader loses connection to the Followers or the Followers are down, the Leader cannot "commit" the journal. 3. BDBJE Internal Failure: * The underlying Berkeley DB (BDBJE) environment may have encountered a fatal error (e.g., 
EnvironmentFailureException
). Check the 
fe.log
or 
bdb/je.info.0
file for earlier exceptions. * Clock Skew: If the system clocks between FE nodes are out of sync by more than 5 seconds, the BDBJE replication group may fail to function. 4. Heavy Metadata Pressure: * A massive burst of metadata operations (e.g., thousands of simultaneous partition creations or frequent small load jobs) can sometimes overwhelm the journal writing thread. Troubleshooting Steps 1. Check Disk Space: Run 
df -h
on the Leader FE node to ensure the 
meta_dir
disk is not at 100% capacity. 2. Inspect Full Logs: Search the 
fe.log
specifically for keywords like 
Exception
, 
EnvironmentFailureException
, or 
ReplicaWriteException
occurring just before the "failed to commit" message. 3. Check FE Status: Run 
SHOW FRONTENDS;
(if the cluster is still somewhat responsive) or check the process status of all FE nodes to see if Followers are healthy. 4. Verify Clock Sync: Ensure NTP or Chrony is running and that all FE nodes have synchronized system times. 5. Check for Hardware/OS Errors: Look at 
dmesg
or system logs for disk I/O errors. Note: If the FE has exited due to this error, simply restarting the Leader FE often resolves transient issues (like a brief network glitch), but if the root cause is a full disk or corrupted BDBJE environment, it will fail again shortly after. References • administration/management/FE_configuration.md
h
@Rocky What does this error mean from FE?
Copy code
ERROR (thrift-server-pool-3578|5621) [SRTThreadPoolServer$WorkerProcess.run():319] Thrift Error occurred during processing of message.
r
This error message 
ERROR (...) [SRTThreadPoolServer$WorkerProcess.run():319] Thrift Error occurred during processing of message
typically indicates a low-level communication failure between the FE (Front End) and another component, such as a BE (Back End), another FE, or a client application. 
SRTThreadPoolServer
is StarRocks' internal implementation of a Thrift server pool. When this error occurs, it means a worker thread encountered an exception while trying to read, process, or write a Thrift message over the network. Common Causes 1. Network Connection Reset/Closed: * The most frequent cause is that the client (the node on the other end) closed the connection prematurely while the FE was still processing the request or trying to send a response. * This can happen due to network instability or a process on the other end crashing. 2. Timeouts: * If the processing takes longer than the configured Thrift timeout, the connection might be dropped. * Check the FE configuration: 
thrift_client_timeout_ms
(default is usually 5000ms). 3. Message Size Limits: * If the request or response payload is exceptionally large, it might exceed the internal buffer limits of the Thrift server. * Check the FE configuration: 
thrift_rpc_max_body_size
. 4. Resource Exhaustion: * If the FE is under heavy load and the number of concurrent Thrift requests exceeds 
thrift_server_max_worker_threads
, new requests might be queued or rejected, leading to connection issues that manifest as Thrift errors. Troubleshooting Steps * Check BE Logs: Look at the 
<http://be.INFO|be.INFO>
or 
be.WARNING
logs around the same timestamp to see if there are corresponding "RPC failed" or "Connection refused" errors. * Check Network Latency: Ensure there is stable connectivity between your FE and BE nodes. * Review FE Load: Check if the FE JVM is experiencing long GC pauses (check 
fe.gc.log
), which can cause Thrift heartbeats or processing to time out. * Validate Configurations: * Verify if 
thrift_server_max_worker_threads
(default 4096) is sufficient for your concurrency. * If you are dealing with very large metadata or complex queries, you might need to increase timeout settings. If this error happens only occasionally, it is often a transient network glitch or a client disconnecting. However, if it is frequent, it usually points to a more persistent network or resource bottleneck. References • faq/Deploy_faq.md • administration/Meta_recovery.md • developers/code-style-guides/thrift-guides.md • knowledge/usage/best-practice.md • administration/management/FE_configuration.md • administration/management/monitoring/metrics.md
h
@Rocky What could have caused this error on BE?
Copy code
W20260224 16:39:00.867549 126470952314432 engine_clone_task.cpp:180] Fail to load snapshot:Rpc error: Couldn't open transport for 100.100.100.100:9060 (socket(): Too many open files)
r
The error message 
socket(): Too many open files
indicates that the StarRocks Backend (BE) process has reached the operating system's limit for the maximum number of open file descriptors (FDs). What happened? In StarRocks, the BE process needs to open many file descriptors for: 1. Data files: Each tablet and its underlying segments are stored as files. 2. Network sockets: Communication between BEs (like the RPC call in your error) and communication between FE and BE require sockets. The specific error occurred during an 
engine_clone_task
. This task is responsible for cloning or migrating tablet replicas between BE nodes (e.g., during rebalancing or when a node is added/recovered). Because cloning involves transferring data, it opens multiple files and network connections simultaneously, which likely pushed the FD count over the system's configured limit. Recommended Fix You need to increase the 
ulimit
for file descriptors on all BE nodes. StarRocks recommends setting the limit to 655350. 1. Temporary Fix (for the current session) You can increase the limit for the current shell session (though you'll need to restart the BE process from this shell):
Copy code
bash
ulimit -n 655350
2. Permanent Fix (Recommended) Modify the system configuration to ensure the limit persists after a reboot. Edit 
/etc/security/limits.conf
and add the following lines:
Copy code
text
* soft nofile 655350
* hard nofile 655350
* soft nproc 65535
* hard nproc 65535
If you are using systemd to manage the StarRocks service, you may also need to add the limit to the service file (e.g., 
/etc/systemd/system/starrocks-be.service
):
Copy code
ini
[Service]
LimitNOFILE=655350
Then run 
systemctl daemon-reload
and restart the BE. How to Verify 1. Check OS limit: Run 
ulimit -n
. 2. Check the running BE process limit: Find the PID of your BE and check its specific limits:
Copy code
bash
   cat /proc/$(pgrep starrocks_be)/limits | grep "Max open files"
If the limit is already high (e.g., 655,350) and you still see this error, it might indicate an FD leak or an extremely high number of tablets/segments on a single BE, but in 99% of cases, it is simply an insufficiently configured 
ulimit
. References • deployment/environment_configurations.md • administration/management/BE_configuration.md • knowledge/trouble_shooting/fe_memory_problems.md • faq/fe_mem_faq.md • administration/Meta_recovery.md • administration/management/logs.md
h
@Rocky What could have caused FE high CPU usage, resulting in all FE nodes crashed?
r
High CPU usage on the StarRocks Frontend (FE) followed by a cluster-wide FE crash is typically caused by resource exhaustion or intensive metadata operations. When the Leader FE crashes, Follower nodes attempt to take over; if they encounter the same bottleneck or "poisonous" metadata, they can also crash in succession. Here are the most common causes and how to investigate them: 1. Frequent Full Garbage Collection (Full GC) This is the most frequent cause of high FE CPU. When the Java heap memory is nearly full, the JVM spends almost all CPU cycles attempting to reclaim memory. * Cause: Large metadata (millions of tablets/partitions), heavy query planning for complex SQL, or memory leaks. * Symptom: You will see 
transfer FE type from LEADER to UNKNOWN. exit
in 
fe.log
. This happens because the JVM "Stop-the-World" pause during Full GC makes the Leader lose its heartbeat with other nodes. * Action: Check your GC logs (
fe.gc.log
). If pauses exceed several seconds, increase the FE heap size (
-Xmx
) in 
fe.conf
. 2. Intensive Metadata Operations Metadata management is CPU-intensive. Specific tasks can spike CPU: * Image Checkpointing: Before StarRocks v3.4, the Leader FE performed image serialization locally. This consumes significant CPU and memory. * Tablet Reports: BE nodes periodically report the status of all tablets to the FE Leader. In large clusters (millions of tablets), this single-threaded processing can overwhelm the FE. * Metadata Replay: If there is a massive burst of "edit logs" (e.g., thousands of 
ALTER TABLE
or partition creations), Followers may spike in CPU as they struggle to replay these logs to stay in sync. 3. Query Planning & High Concurrency The FE is responsible for parsing, analyzing, and optimizing SQL. * High Concurrency: A sudden surge in client connections can exhaust the FE thread pool and CPU. * Complex SQL: Queries with thousands of 
IN
clauses or hundreds of joins require significant CPU during the planning phase. * Audit Logs: Writing extremely high volumes of audit logs to a slow disk can cause backpressure, leading to thread blocking and high CPU. 4. BDBJE Latch Timeouts (Disk I/O) StarRocks uses BDBJE for metadata replication. * Cause: If the metadata disk (specified by 
meta_dir
) is slow or has high I/O wait, BDBJE can experience "Latch timeouts" (
LogbufferPool_FullLatch
). * Result: This can cause the FE process to become unresponsive and eventually crash. Ensure your FE metadata is stored on high-performance SSDs. 5. JVM/Environment Issues * OOM Killer: If the FE process exceeds the physical RAM available on the machine (Heap + Off-heap memory), the Linux kernel may kill the process. Check 
dmesg
or 
/var/log/messages
for "Out of memory: Kill process". * MALLOC_ARENA_MAX: In some environments, the default glibc memory allocation can lead to virtual memory bloat. Setting 
export MALLOC_ARENA_MAX=1
in 
fe.conf
is a common fix. Troubleshooting Steps: 1. Check FE Logs: Look for 
Full GC
, 
warn
or 
error
messages in 
fe.log
and 
fe.warn.log
. 2. Analyze GC Logs: Use tools like gceasy.io to analyze 
fe.gc.log.0.current
. 3. Capture Jstack: If the FE is still running but CPU is high, run 
jstack <pid> > jstack.out
to see which threads are busy. 4. Review Metadata Stats: Check the number of tablets and partitions. If you have millions of tablets, consider increasing the FE heap or merging small partitions. 5. Audit Log Check: Check the 
fe/log/fe.audit.log
to see if there was a spike in slow or complex queries just before the crash. References • knowledge/trouble_shooting/fe_memory_problems.md • faq/fe_mem_faq.md
h
@Rocky SHOW SNAPSHOT shows Invalid: __info_xxx.part. What does the error mean and what could have caused it?
r
The error 
Invalid: __info_xxx.part
in the output of 
SHOW SNAPSHOT
typically indicates that StarRocks found a metadata file in your backup repository that is incomplete or corrupted. What the error means: When you perform a BACKUP, StarRocks creates metadata files in the repository (such as S3, HDFS, or Azure Blob Storage) to describe the snapshot. These files are often named 
__info_<snapshot_name>
. The 
.part
suffix is a temporary extension used during the upload process. Once the file is successfully and completely written to the storage, StarRocks renames it to the final version (removing the 
.part
). If 
SHOW SNAPSHOT
displays 
Invalid: __info_xxx.part
, it means the system detected a partial metadata file that was never finalized. Potential Causes: 1. Interrupted Backup Job: The most common cause is that a 
BACKUP
task was interrupted before it could finish. This could be due to: • An FE (Frontend) restart or crash during the backup process. • Network instability between StarRocks and the storage repository. • The backup job being manually cancelled while it was in the "finalizing" stage. 2. Storage Write Failures: If the underlying storage (e.g., S3) experienced a timeout or a write error at the very end of the backup process, the temporary 
.part
file might remain in the bucket without being renamed. 3. Manual File Manipulation: If files in the repository were manually moved, deleted, or modified, it could leave the repository in an inconsistent state. How to resolve it: • Check Job Status: Run 
SHOW BACKUP
to see if there are any failed or cancelled jobs that correspond to that snapshot name. • Clean up: If the snapshot is indeed invalid and you don't need it, you can manually delete the 
.part
file from your storage repository (e.g., using 
aws s3 rm
or 
hdfs dfs -rm
). • Retry Backup: If the snapshot is required, you will need to run the 
BACKUP
command again to create a complete and valid snapshot. If you continue to see this error despite successful backup jobs, ensure that your FE has stable network connectivity and the necessary permissions to perform 
RENAME
or 
MOVE
operations on the storage service.
h
@Rocky I am running a backup but it is stuck at State: PENDING. What might have caused this?
r
Hi there! I'm Rocky. Sorry to hear your backup is stuck. When a StarRocks 
BACKUP
operation stays in the PENDING state, it usually means the job has been submitted to the Front End (FE) but hasn't successfully transitioned into the active snapshotting phase. Here are the most common reasons this might happen: 1. Repository Connectivity/Configuration: The FE might be having trouble communicating with your backup repository (e.g., S3, HDFS, or MinIO). If the repository is unreachable or the credentials/permissions are incorrect, the job may hang in PENDING while it retries. * Action: Double-check your 
CREATE REPOSITORY
statement and ensure the FE and all BEs have network access to the storage destination. 2. Snapshot Preparation: In the PENDING stage, the FE sends snapshot requests to the relevant Back Ends (BEs). If the BEs are under heavy load or if there is a high frequency of data ingestion, it might take longer to initialize the consistent snapshots. 3. Concurrent Jobs: StarRocks typically allows only one backup or restore job per database at a time. If there is another job running or if a previous job didn't clean up properly, the new job will wait. * Action: Run 
SHOW BACKUP;
to see if there are other tasks and check their status. 4. FE Task Scheduling: Sometimes internal FE threads responsible for scheduling these tasks are busy or hitting a limit. Troubleshooting Steps: * Check Detailed Status: Run the following command to see if there's an error message hidden in the job details:
Copy code
sql
    SHOW BACKUP FROM your_database_name;
* Inspect FE Logs: Look into the 
fe.log
and 
fe.warn.log
on your leader FE. Search for keywords like "BackupJob" or the specific ID of your backup job. This is usually the quickest way to find the root cause (e.g., "Access Denied" or "Timeout"). * Check Repository: Test if you can manually access the storage from the StarRocks nodes using a CLI tool (like 
aws s3 ls
or 
hadoop fs -ls
). If the job stays stuck for a very long time without any log activity, you can try to cancel it and restart:
Copy code
sql
CANCEL BACKUP FROM your_database_name;
Let me know if you see any specific errors in the logs!
h
@Rocky How to find logs about streamload?
r
To find logs and information about Stream Load in StarRocks, you can check several different levels depending on whether you are looking for data quality errors, job history, or system-level debugging. 1. Data Quality Error Logs (
ErrorURL
) If a Stream Load job completes with a 
Fail
status or has 
NumberFilteredRows > 0
, the JSON response will include an 
ErrorURL
. * How to view: Copy the 
ErrorURL
from the response and use 
curl
or a browser to view it. * What it shows: Specific rows that were filtered out and the reason (e.g., "column count mismatch" or "invalid data format"). 2. Information Schema (System Tables) Starting from StarRocks v3.0, you can query metadata and logs directly via SQL: * 
information_schema.stream_loads
: Provides the status, error messages, and tracking URLs for recent Stream Load jobs.
Copy code
sql
    SELECT * FROM information_schema.stream_loads WHERE LABEL = 'your_label';
* 
information_schema.load_tracking_logs
: Provides more detailed error tracking for specific jobs.
Copy code
sql
    SELECT * FROM information_schema.load_tracking_logs WHERE LABEL = 'your_label';
3. Detailed HTTP Verbose Logs If you need to see the exact HTTP requests and responses for debugging (e.g., headers, body info), you can enable verbose logging on the Backend (BE). * BE Configuration: 
enable_stream_load_verbose_log
(Default: 
false
). * How to enable:
Copy code
sql
    ADMIN SET FRONTEND CONFIG ("enable_stream_load_verbose_log" = "true"); -- For FE
    -- Or for BE via curl:
    curl -X POST http://<be_ip>:<be_http_port>/api/update_config?enable_stream_load_verbose_log=true
* Where to find them: These logs will appear in the 
<http://be.INFO|be.INFO>
log file on the specific BE node that handled the request. 4. System Log Files If a load fails due to a system crash or unexpected error (not a data quality issue), check the standard log files: * FE Logs (
fe.log
/ 
fe.warn
): Shows the initial coordination and transaction beginning/committing. Search for your 
Label
or 
TxnId
. * BE Logs (
<http://be.INFO|be.INFO>
/ 
be.WARNING
): Shows the actual data processing and writing to tablets. Search for the 
TxnId
returned in the Stream Load response. 5. Summary Tips * Identify the Label: Always use a unique 
Label
for your Stream Load jobs; it makes searching through logs and system tables much easier. * Search by TxnID: The 
TxnId
is the most reliable way to trace a specific load across FE and BE logs. References • administration/management/monitoring/metrics.md • loading/StreamLoad.md • administration/management/BE_configuration.md • sql-reference/sql-statements/loading_unloading/STREAM_LOAD.md • sql-reference/information_schema/load_tracking_logs.md • sql-reference/information_schema/stream_loads.md
h
@Rocky How to find logs about streamload about read timeout?
r
To find logs related to Stream Load read timeouts in StarRocks, you should look into both the Backend (BE) and Frontend (FE) log files. Since Stream Load is a two-step process (FE redirects to BE, then BE processes the data), the most detailed information is usually on the BE. 1. Where to Look for Logs * BE Logs (Most Important): Look in the 
${STARROCKS_HOME}/log/
directory of the BE node that handled the request. * 
<http://be.INFO|be.INFO>
: This is the primary runtime log. You can search for your Stream Load 
Label
or 
TxnId
here to see the execution details. * 
be.WARNING
: If the load timed out or failed due to a network/IO issue, it will likely be recorded here. * FE Logs: Look in 
${STARROCKS_HOME}/log/
on the FE node. * 
fe.log
: Useful for seeing the initial request, redirection, and whether the FE marked the transaction as "Aborted" due to timeout. * Error Log URL: If the load finished but failed with errors (e.g., data quality), check the 
ErrorURL
provided in the JSON response of the Stream Load. You can also find these in 
${STARROCKS_HOME}/storage/error_log/
on the BE. 2. How to Search the Logs When searching, use the Label you assigned to the load or the TxnId (Transaction ID) returned in the response. Example command on BE:
Copy code
bash
grep "your_load_label" <http://be.INFO|be.INFO>
# or search for timeout specifically
grep -i "timeout" <http://be.INFO|be.INFO> | grep "StreamLoad"
3. Common Log Patterns for Read Timeout If you encounter a "read timeout," look for these keywords: * `timeout`: General timeout occurred. * `deadline exceeded`: The internal RPC or execution deadline was reached. * 
Broken pipe
or `Connection reset`: Often indicates the client closed the connection before the BE finished reading the data. * `stream_load_default_timeout_second`: Indicates the job hit the default system timeout. 4. Useful Configurations for Debugging If the logs are not detailed enough, you can enable verbose logging for Stream Load on the BEs: * Enable Verbose HTTP Logs:
Copy code
sql
    -- Run this on the BE or via SQL (if supported by your version)
    -- to log HTTP requests and responses for Stream Load.
    ADMIN SET FRONTEND CONFIG ("enable_stream_load_verbose_log" = "true"); -- (Actually a BE config, set via BE HTTP API or be.conf)
Note: In newer versions, you can set this dynamically on BE via the HTTP API: 
curl -X POST <http://be_ip:be_http_port/api/update_config?enable_stream_load_verbose_log=true>
5. Parameters to Consider If you find that your loads are timing out frequently, you might need to adjust: * 
stream_load_default_timeout_second
(FE): Default is 600s. Increase this if your data is large. * 
streaming_load_rpc_max_alive_time_sec
(BE): The RPC timeout for Stream Load processes. * Client-side timeout: Ensure the tool you are using (e.g., 
curl --connect-timeout
or an SDK) has a longer timeout than the StarRocks server-side timeout. References • administration/management/FE_configuration.md • administration/management/BE_configuration.md • faq/loading/Broker_load_faq.md • faq/loading/Loading_faq.md
h
@Rocky What does this error mean what could be the cause?
Copy code
waiting secondary replicas too long, load_id: , txn_id: , timeout: 7181485 ms, elapsed time: 30001 ms, primary replica host: , num finished tablets: 0, num unfinished tablets: 2342, last unfinished tablet: [tablet_id: , state 1]
r
This error message, 
waiting secondary replicas too long
, typically occurs when loading data into a Primary Key table in StarRocks. What it means In StarRocks Primary Key tables, the loading process uses a "Primary-Secondary" replication model. The Primary Replica (a specific BE) receives the data, processes it, and then sends the data segments to the Secondary Replicas (other BEs). The Primary Replica then waits for the Secondary Replicas to acknowledge that they have successfully received and written the data. If the Secondary Replicas do not respond within an internal timeout window—which is 30 seconds (as seen in your 
elapsed time: 30001 ms
)—the load job fails with this error. Even if your total load timeout (
timeout: 7181485 ms
) is very high, this internal synchronization check has a much shorter limit to prevent the system from hanging. Possible Causes 1. Overloaded BEs: The most common cause is that one or more Backend (BE) nodes are under heavy pressure (CPU, I/O, or memory). The secondary replicas are too busy to process the replication request and acknowledge it within 30 seconds. 2. Too many tablets in one transaction: Your error shows 
num unfinished tablets: 2342
. This is a very high number of tablets to process in a single synchronization step. If a load job touches too many buckets or partitions at once, it can overwhelm the replication process. 3. Primary Key Index Overhead: For Primary Key tables, every write requires a primary key index lookup and update. If the index is large or the BEs are struggling with memory, this process slows down, causing the replication acknowledgment to time out. 4. Network Congestion: Slow or unstable network connectivity between BE nodes can delay the transmission of segments or the acknowledgment signals. 5. Disk I/O Bottleneck: If the disks on the secondary BEs are experiencing high latency (e.g., due to heavy compaction or other I/O intensive tasks), they cannot write the data segments quickly enough. Recommended Actions * Reduce Load Concurrency/Batch Size: If you are using Routine Load or Stream Load, try reducing the batch size or the frequency of the loads. This reduces the number of tablets the system has to synchronize at once. * Check BE Resource Usage: Monitor your BE nodes' CPU and Disk I/O. Use 
top
or StarRocks' Grafana dashboards to see if any nodes are hitting 100% utilization. * Optimize Tablet Count: If your table has too many buckets (
BUCKETS
in the 
DISTRIBUTED BY
clause) or too many partitions, consider reducing them to decrease the "width" of the load. * Adjust BE Configurations: * 
transaction_apply_worker_count
: Increase this in 
be.conf
(default is usually the number of CPU cores) to allow more threads to handle Primary Key update applications. * 
tablet_max_versions
: Check if your tables have a high compaction score. If compaction is lagging, metadata operations become slower. * Check Persistent Index: Ensure that 
enable_persistent_index
is set to 
true
for your Primary Key table to reduce memory pressure on the BEs. References • administration/management/BE_configuration.md
h
@Rocky What does this error mean and what could have caused it
Copy code
async_delta_writer.cpp:187] Fail to execution_queue_execute: 22
r
The error message 
async_delta_writer.cpp:187] Fail to execution_queue_execute: 22
indicates that a StarRocks Backend (BE) process failed to submit a data writing task to the internal asynchronous execution queue. What the error means * 
async_delta_writer
: This is a component in the BE responsible for handling data writes (memtable flushes) asynchronously during data loading (e.g., Stream Load, Broker Load, or Routine Load). * 
execution_queue_execute
: This is the function call that attempts to place a write task into a thread pool queue for execution. * 
22
: This is the error code. In Linux/POSIX systems, 
22
stands for 
EINVAL
(Invalid Argument). In the context of StarRocks task queues, this specifically means the queue is no longer accepting tasks, usually because it has been closed or is shutting down. Common Causes 1. BE Shutdown or Restart: This is the most frequent cause. If the BE process is in the middle of a restart or shutdown, it closes its execution queues. Any pending or new write tasks will fail with error 22. 2. Load Job Cancellation or Timeout: If a load job (like a Stream Load) is cancelled by the user or times out on the Frontend (FE), the FE tells the BE to stop. The BE then closes the writers for that job. If a write thread tries to submit one last task to a closing writer, you will see this error. 3. Tablet Migration or Deletion: If the tablet being written to is currently being migrated to another BE or has been dropped (e.g., the table or partition was deleted), the local writer is closed, leading to this failure. 4. High Resource Pressure: In rare cases, extreme memory or CPU pressure can cause the BE to abort certain tasks or close writers to prevent a crash, though this is usually accompanied by OOM (Out of Memory) logs. Troubleshooting Steps * Check BE Uptime: Run 
ps -ef | grep starrocks_be
or check the BE's 
be.out
log to see if the process recently restarted. * Check Load Job Status: If you are using Stream Load or Routine Load, check if the job state is 
CANCELLED
or if it failed with a 
TIMEOUT
.
Copy code
sql
    SHOW LOAD; -- For Stream/Broker Load
    SHOW ROUTINE LOAD; -- For Routine Load
* Monitor BE Logs: Look at the 
<http://be.INFO|be.INFO>
logs around the same timestamp for messages like 
stopping
, 
shutting down
, or 
tablet is being dropped
. * Adjust Concurrency (if happening frequently under load): If this happens during high-concurrency loads without restarts, consider checking the BE configuration 
number_tablet_writer_threads
. The default is half of the CPU cores. If the system is overloaded, you may need to reduce the loading frequency or scale the cluster. In most cases, if this happens only once or twice during a service maintenance window, it can be safely ignored as it's a symptom of a clean shutdown or a timed-out job. References • sql-reference/Error_code.md • developers/code-style-guides/restful-api-standard.md
h
@Rocky How to find query by txn_id?
r
To find the query associated with a specific 
txn_id
(Transaction ID) in StarRocks, you can use the following methods depending on the type of operation (e.g., 
INSERT
or 
STREAM LOAD
). 1. Use 
SHOW TRANSACTION
 The most direct way to get details for a specific transaction is the 
SHOW TRANSACTION
command. This will return the Label associated with the transaction.
Copy code
sql
SHOW TRANSACTION WHERE ID = <txn_id>;
* For 
INSERT
statements: The 
Label
is typically formatted as 
insert_<query_id>
. You can extract the 
query_id
from this string. * For Stream Loads: The 
Label
is the one provided by the user during the load request. 2. Query 
information_schema
Views (v3.1+) If you are on StarRocks v3.1 or later, you can use the 
information_schema.loads
view. This view contains a 
RUNTIME_DETAILS
column which includes the 
txn_id
.
Copy code
sql
-- Find the label and type for a specific transaction ID
SELECT LABEL, TYPE, STATE 
FROM information_schema.loads 
WHERE RUNTIME_DETAILS LIKE '%"txn_id": <txn_id>%';
For Stream Loads specifically, there is a dedicated view with a direct 
TXN_ID
column:
Copy code
sql
SELECT LABEL, DB_NAME, TABLE_NAME, STATE 
FROM information_schema.stream_loads 
WHERE TXN_ID = <txn_id>;
3. Find the SQL via Query ID Once you have retrieved the 
query_id
(usually from the 
insert_<query_id>
label), you can find the actual SQL text using one of these methods: * Audit Log: Search for the 
query_id
in the 
fe.audit.log
file on your FE node. * Audit Table: If you have the AuditLoader plugin installed, query your audit table:
Copy code
sql
    SELECT stmt FROM starrocks_audit_db*.starrocks_audit_tbl*
    WHERE queryId = '<query_id>';
* Query Profile: If the query is still in the profile cache, you can view its details:
Copy code
sql
    get_query_profile('<query_id>');
Summary of Workflow 1. Run 
SHOW TRANSACTION WHERE ID = <txn_id>;
. 2. Note the 
Label
. If it looks like 
insert_01234567-89ab-cdef...
, then 
01234567-89ab-cdef...
is your Query ID. 3. Search for that Query ID in your Audit Logs to find the original SQL statement. References • sql-reference/information_schema/be_txns.md • sql-reference/information_schema/loads.md • sql-reference/information_schema/stream_loads.md
h
@Rocky Why streamload could cause high I/O wait on the BE nodes
r
Stream Load can cause high I/O wait on BE nodes for several reasons, primarily related to how StarRocks handles data ingestion and background maintenance. The most common causes are: 1. High-Frequency Small Loads (The "Small File" Problem) If you are performing many small Stream Loads (e.g., loading every few seconds or small batches of rows), it creates a large number of small data files called Rowsets. • Compaction Pressure: To maintain query performance, StarRocks must merge these small files into larger ones through a background process called Compaction. Compaction is extremely I/O intensive because it reads multiple files, merges them in memory, and writes new files back to disk. • Write Amplification: Frequent small loads trigger aggressive "Cumulative Compaction," which can saturate disk I/O bandwidth. 2. MemTable Flushing Data from a Stream Load is first buffered in memory (MemTable). Once the MemTable reaches a threshold (default 100MB or based on time), it is flushed to disk. • If you have many concurrent Stream Load jobs, multiple BE threads will attempt to flush data to disk simultaneously. • If the disk subsystem (especially HDDs) cannot keep up with the write throughput, threads will enter an I/O wait state. 3. Replica Synchronization StarRocks typically maintains multiple replicas (default 3) for data reliability. • During a Stream Load, the data must be written to the disks of all relevant BE nodes. • If one BE node has slower disks or higher load, the coordination of these synchronous writes can lead to increased latency and perceived I/O wait across the involved nodes. 4. Hardware Limitations • HDD vs. SSD: StarRocks is optimized for high-throughput environments. If BE nodes use HDDs, the random I/O generated by simultaneous loading and background compaction will quickly lead to high I/O wait. • Disk Saturation: If the total ingestion rate + compaction rate exceeds the physical MB/s limit of your disks, I/O wait is inevitable. ────────── How to Diagnose and Fix 1. Check Compaction Status Run the following command to see if compaction is struggling to keep up:
Copy code
sql
SHOW PROC '/compactions';
Look at the 
CompactionScore
. If it is consistently high (e.g., > 100), compaction is falling behind, which is likely causing the I/O spikes. 2. Analyze Load Profiles Enable the load profile for your table to see where the time is being spent:
Copy code
sql
ALTER TABLE your_table SET ('enable_load_profile' = 'true');
After a load, use 
SHOW PROFILELIST
and 
ANALYZE PROFILE
to check metrics like 
WaitFlushTime
. A high 
WaitFlushTime
confirms that the system is waiting on the disk to finish writing. 3. Optimization Recommendations * Increase Batch Size: Instead of many small loads, aggregate data on the client side and send larger batches (e.g., 100MB to 1GB per load) every 10–60 seconds. * Use SSDs: It is highly recommended to use NVMe or SATA SSDs for BE storage in production. * Tune Flush Threads: If you have high-performance disks but still see I/O wait, you can increase the number of flush threads in `be.conf`:
Copy code
conf
    flush_thread_num_per_store = 4  # Default is often 2
* Tune Compaction: Reduce the number of segments merged at once to smooth out I/O spikes:
Copy code
conf
    max_cumulative_compaction_num_singleton_deltas = 100 # Default is 500
References • loading/loading_introduction/troubleshooting_loading.md
h
@Rocky What does this error from BE mean and what could have caused it?
Copy code
Failed to sync segment SyncChannnel [host: , port: , load_id: a72ydhftser, tablet_id: 11111, txn_id: 22222] err Internal error: PTabletWriterAddSegmentRequest tablet_id 11111 not exists
r
This error typically occurs during the data loading process (such as Broker Load, Stream Load, or Routine Load) when the system is transferring data segments between Backend (BE) nodes. What the error means The error 
PTabletWriterAddSegmentRequest tablet_id 11111 not exists
means that a BE node received an internal request to add a data segment to tablet 
11111
, but that BE does not have an active "Tablet Writer" for that specific tablet under the given 
load_id
and 
txn_id
. In StarRocks, when data is loaded, it is often shuffled between BEs so that the BE holding the tablet replica can write the data. If the receiving BE hasn't initialized a writer for that tablet or has already closed it, this error is thrown. Common Causes 1. Concurrent Load Failure: If another part of the same load job failed on a different tablet or BE, the entire transaction might have been aborted. The BE then cleans up its writers. If a "late" segment arrives from another node for that same transaction, the BE reports it "not exists." 2. Load Timeout: The load job may have exceeded its timeout (e.g., 
streaming_load_rpc_max_alive_time_sec
or the session variable 
query_timeout
). Once the timeout is reached, the BE closes the writers to free up resources. 3. Tablet Migration or Rebalancing: If a tablet was moved to a different BE (due to decommissioning a node or cluster balancing) while the load was in progress, the original BE will no longer recognize the tablet as its own. 4. BE Restart: If the destination BE restarted during the load, it would lose the in-memory state of the active load transaction. 5. Schema Change: Running a 
SCHEMA CHANGE
or 
ROLLUP
on the same table during a load can sometimes lead to tablet version mismatches or writer closures. How to Troubleshoot * Check BE Logs: Look at the 
<http://be.INFO|be.INFO>
or 
be.WARNING
logs on the BE that reported the error. Search for the 
load_id
(
a72ydhftser
in your case) to see if there was an earlier error that caused the writers to close (e.g., "memory limit exceeded," "disk full," or "RPC timeout"). * Verify Tablet Location: Run the following SQL to see which BEs currently hold the replicas for that tablet:
Copy code
sql
    SHOW TABLET 11111;
Compare the 
BackendId
from the output with the host that threw the error. * Check Load Status: If this was a Broker Load or Routine Load, check the job status:
Copy code
sql
    SHOW LOAD WHERE LABEL = 'your_label';
    -- or for Routine Load
    SHOW ROUTINE LOAD FOR your_table;
* Increase Timeouts: If the error is frequent during large loads, try increasing the timeout settings: * In FE (session variable): 
SET query_timeout = 3600;
* In `be.conf`: 
streaming_load_rpc_max_alive_time_sec
(default is usually 600s). If the error only happens once, it is likely a transient network or timing issue. If it persists, it usually points to a resource bottleneck (memory/disk) or a version mismatch. References • faq/loading/Loading_faq.md • sql-reference/sql-statements/table_bucket_part_index/SHOW_TABLET.md • faq/operation_maintenance_faq.md • sql-reference/information_schema/be_tablets.md • table_design/table_types/primary_key_table.md
h
@Rocky What does this error mean from BE and what could have caused it?
Copy code
W20260315 05:10:45.727612 12337242 segment_replicate_executor.cpp:189] Failed to send rpc to SyncChannnel [host: , port: , load_id: 529d17c, tablet_id: 11111, txn_id: 22222] err=Internal error: PTabletWriterAddSegmentRequest tablet_id 11111 not exists
r
The error message you're seeing occurs during Single Replica Load (also known as Segment Replication). In this mode, instead of every replica receiving and processing the data independently, only one "primary" replica receives the data. Once it writes a data segment to disk, it replicates that physical segment file to the other "follower" replicas using an internal RPC called 
PTabletWriterAddSegmentRequest
. What the error means The error 
Internal error: PTabletWriterAddSegmentRequest tablet_id 11111 not exists
means that the Primary BE tried to send a data segment to a Follower BE, but the Follower BE reported that it does not have tablet 11111 in its local metadata. Potential Causes 1. Tablet Migration or Balancing: StarRocks might have moved the tablet replica from that specific BE to another BE while the load job was still running. The load process is still trying to send data to the "old" location based on the replica map it received at the start of the transaction. 2. Table/Partition Drop or Truncate: If a user dropped or truncated the table/partition while the data was being loaded, the BE would have deleted the local tablet metadata, leading to this "not exists" error when the load tries to finish writing. 3. Stale Metadata on the Coordinator/Primary BE: The BE acting as the coordinator or the primary for the load might have an outdated view of where the replicas are located. This can happen if there was a recent cluster change (e.g., adding/removing nodes) and the FE hasn't propagated the new replica locations to all BEs yet. 4. BE Restart or Storage Issues: If the target BE recently restarted, it might still be loading its metadata into memory. Alternatively, if there was a disk error that caused the tablet to be marked as "bad" or removed, the RPC would fail. 5. Schema Change: An ongoing schema change might have triggered a tablet conversion or migration, causing the original tablet ID to become invalid or be replaced. How to Troubleshoot * Check Tablet Status: Run the following command to see where the replicas for that tablet are currently located:
Copy code
sql
    SHOW TABLET 11111;
Compare the 
BackendId
in the output with the Backend that reported the error. If the Backend is no longer listed as a host for that tablet, it confirms a migration or deletion occurred. * Check Cluster Balance Tasks: See if there are any active tablet migrations that might be causing replicas to move:
Copy code
sql
    SHOW PROC '/cluster_balance/running_tablets';
* Verify if Single Replica Load is enabled: This error is specific to this feature. You can check your global or session settings:
Copy code
sql
    SHOW VARIABLES LIKE '%enable_single_replica_load%';
* Identify the Table: If you don't know which table is affected, you can find it using the 
tablet_id
from the error:
Copy code
sql
    SELECT * FROM information_schema.be_tablets WHERE tablet_id = 11111;
Summary: This is usually a transient error caused by the cluster dynamically moving data (balancing) while a load is in progress. If the load job fails, StarRocks will typically retry it automatically. If it happens frequently, you may need to look into why your tablets are migrating so often or check for disk health issues on the BEs. References • administration/management/resource_management/Replica.md
h
@Rocky What's the recommended specs (iops, throughput) for EBS gp3 volume of BE?
r
For StarRocks Backend (BE) nodes using AWS EBS gp3 volumes, the recommended specifications depend on your workload (concurrency, data volume, and table types). Based on official hardware requirements and general best practices for high-performance deployments: 1. Recommended Baseline Specs * IOPS: Minimum 3,000 IOPS (gp3 baseline). For production workloads with intensive random I/O (like Primary Key tables or high-concurrency queries), it is common to provision 5,000 to 10,000+ IOPS. * Throughput: Minimum 125 MB/s (gp3 baseline). To avoid bottlenecks during large-scale scans, data loading, or compaction, it is highly recommended to increase this to 250 MB/s - 500 MB/s. 2. Why High Storage Specs Matter * Primary Key Tables: If you use the Primary Key model with the persistent index enabled, SSD-level performance is mandatory. Low IOPS will significantly slow down data ingestion and point lookups. * Compaction: StarRocks performs background "compaction" to merge data files. Insufficient disk throughput can cause compaction to fall behind, leading to "too many versions" errors and degraded query performance. * Data Scans: Since StarRocks is a columnar database designed for massive parallel processing, your disk throughput is often the first bottleneck during large aggregation queries. 3. Key Configuration Best Practices * File System: Use XFS (highly recommended over ext4) for better performance with StarRocks' I/O patterns. * Disk Count: For high-throughput requirements, you can attach multiple EBS volumes to a single BE. StarRocks can utilize multiple storage paths (configured via 
storage_root_path
in 
be.conf
) to parallelize I/O across volumes. * Mount Options: Ensure you use 
noatime
in your mount settings to reduce unnecessary write operations. Summary Table for gp3 Workload Level | IOPS | Throughput Development/Testing | 3,000 (Baseline) | 125 MB/s (Baseline) Standard Production | 5,000 - 8,000 | 250 MB/s High Performance | 10,000 - 16,000 | 500 MB/s - 1,000 MB/s Rocky's Tip: Always monitor the 
be_bytes_read_per_second
and 
be_bytes_written_per_second
metrics to see if your workload is hitting the EBS limits. If you see high "Disk Utility" or "I/O Wait" in your monitoring, consider scaling up the gp3 throughput first, as it is often more cost-effective than adding more IOPS. References • faq/operation_maintenance_faq.md • knowledge/usage/load/shared_data_batch_ingestion.md • faq/Others.md • knowledge/usage/best-practice.md • quick_start/routine-load.md • deployment/deployment_prerequisites.md
h
@Rocky how to find be_bytes_written_per_second and 
be_bytes_read_per_second?
r
To find the 
be_bytes_written_per_second
and 
be_bytes_read_per_second
metrics in StarRocks, you can use the following methods: 1. Directly from the BE Metrics Endpoint Each Backend (BE) node exposes a metrics endpoint in Prometheus format. You can access it via a web browser or `curl`:
Copy code
bash
curl http://<BE_IP>:<BE_HTTP_PORT>/metrics | grep be_bytes_
* Default BE HTTP Port: 
8040
* Metrics: * `starrocks_be_be_bytes_read_per_second`: The current read speed of the BE (Bytes/s). * `starrocks_be_be_bytes_written_per_second`: The current write speed of the BE (Bytes/s). 2. Using Prometheus/Grafana If you have monitoring set up (highly recommended for production), these metrics are typically collected by Prometheus. * Prometheus Query: Simply search for 
starrocks_be_be_bytes_read_per_second
or 
starrocks_be_be_bytes_written_per_second
in the Prometheus UI. * Grafana: These metrics are usually included in the standard StarRocks BE dashboard under the I/O or Throughput sections. 3. Configuration Requirement For these specific "per second" metrics to be calculated and reported accurately, ensure the following BE configuration is enabled (it is 
true
by default): * 
enable_metric_calculator
: When set to 
true
, the BE launches a background thread to compute derived metrics like bytes/sec every 15 seconds. Summary of the Metrics: Metric Name | Unit | Description 
be_bytes_read_per_second
| Bytes/s | Average read speed of the BE node. 
be_bytes_written_per_second
| Bytes/s | Average write speed of the BE node. References • administration/management/monitoring/metrics.md • administration/management/BE_configuration.md
h
@Rocky What is the sequence to turn off the FE and BE nodes?
r
To turn off StarRocks nodes, the recommended sequence depends on whether you are doing a full cluster shutdown or a rolling upgrade. 1. Full Cluster Shutdown Sequence If you are shutting down the entire cluster, the generally recommended sequence is to stop the workers first and then the management layer: 1. Stop Data Ingestion & Applications: Ensure no new data is being loaded or queried. 2. Stop BE (Backend) / CN (Compute) nodes: Stop the nodes that handle data storage and computation first. 3. Stop FE (Frontend) nodes: Stop the management nodes last. Commands: * BE: 
cd <be_dir>/bin && ./stop_be.sh
* FE: 
cd <fe_dir>/bin && ./stop_fe.sh
────────── 2. Rolling Upgrade Sequence If you are turning off nodes one by one to upgrade them (Rolling Upgrade), the sequence is specific to ensure cluster availability: 1. Stop and Upgrade BE/CN nodes first: Upgrade these one by one. StarRocks is designed so that newer BEs/CNs are backward compatible with older FEs. 2. Stop and Upgrade FE nodes last: * Upgrade Follower FE nodes one by one. * Upgrade the Leader FE node last (stopping the Leader will trigger a new leader election among the already upgraded followers). ────────── 3. Graceful Exit (Recommended for v3.3+) Starting from v3.3, StarRocks supports a Graceful Exit mechanism which minimizes impact on running queries. Use the 
-g
flag to initiate this: * FE Graceful Exit:
Copy code
bash
    ./stop_fe.sh -g --timeout 60
This allows the FE to finish existing sessions and tells Load Balancers to stop routing traffic to it. * BE/CN Graceful Exit:
Copy code
bash
    ./stop_be.sh -g --timeout 600
The BE will mark itself as "exiting," reject new query fragments, and wait for in-flight tasks to complete before shutting down. Summary of Order * Shutdown: BEs $\rightarrow$ FEs. * Startup: FEs $\rightarrow$ BEs. * Upgrade: BEs $\rightarrow$ FEs (Followers first, then Leader). References • deployment/upgrade.md • deployment/downgrade.md
h
@Rocky What is the sequence to start a fully shutdown FE and BE nodes?
r
To start a fully shutdown StarRocks cluster, the recommended sequence is to start the Frontends (FE) first, followed by the Backends (BE) or Compute Nodes (CN). 1. Start FE Nodes FE nodes manage the cluster metadata and the control plane. They must elect a leader before the cluster can fully function. * Action: Run the start script on all FE nodes.
Copy code
bash
    # On each FE node
    cd fe/bin/
    ./start_fe.sh --daemon
* Verification: Log in to the leader FE (via a MySQL client) and check the FE status.
Copy code
sql
    SHOW PROC '/frontends'\G
Ensure that one FE has the 
Role
of 
LEADER
and that the 
Alive
status is 
true
for the others. 2. Start BE/CN Nodes Once the FE leader is elected and the metadata is ready, you can start the data nodes. * Action: Run the start script on all BE/CN nodes.
Copy code
bash
    # On each BE node
    cd be/bin/
    ./start_be.sh --daemon

    # On each CN node (if applicable)
    cd cn/bin/
    ./start_cn.sh --daemon
* Verification: From your MySQL client, verify that the nodes are connected and active.
Copy code
sql
    SHOW PROC '/backends'\G
    -- Or for CN nodes:
    SHOW PROC '/compute_nodes'\G
Confirm that the 
Alive
column shows 
true
for all nodes. Important Notes: * Metadata Consistency: You do not need to use the 
--helper
flag when restarting an existing cluster. That flag is only used when adding a new node to the cluster for the first time. * BDBJE Election: In a multi-FE high-availability (HA) setup, FE nodes use BDBJE to elect a leader. If you start the FE nodes one by one, the cluster will not be "Ready" until a quorum (the majority) of FOLLOWER nodes are up and a LEADER is elected. * BE Heartbeats: If you start BE nodes before the FEs are ready, the BEs will simply log heartbeat failures and continue to retry until the FEs become available. References • administration/stargo.md • administration/management/graceful_exit.md • deployment/deploy_manually.md
h
@Rocky What to do if FE heap memory usage is constantly high?
r
High FE heap memory usage can lead to performance degradation, long GC pauses, or even FE service restarts. To troubleshoot and resolve this, you can follow these steps: 1. Immediate Quick Fixes If the FE is in a critical state (e.g., frequently crashing or unresponsive), you can: * Increase JVM Heap Size: Modify the 
-Xmx
parameter in 
fe/conf/fe.conf
. * For machines with < 32GB RAM: Set 
-Xmx
to ~70% of total RAM. * For machines with > 32GB RAM: Set 
-Xmx
to ~80% of total RAM. * Restart FE: A restart provides temporary relief by clearing the heap, but the usage may climb back up if the root cause isn't addressed. 2. Diagnostic Tools & Monitoring To understand what is occupying the heap, use the following tools: A. Memory Profiles (v3.3.6 and later) StarRocks automatically generates memory allocation profiles. * Location: Check 
fe/log/proc_profile/
. * Analysis: Look for 
.tgz
files containing HTML flame graphs. These graphs show which functions or modules are allocating the most memory. A wide bar indicates a high allocation hotspot. B. Memory Usage Tracker (v3.3.7 and later) StarRocks tracks memory per module and logs it periodically. * Usage: Search for "Memory Usage Tracker" in 
fe.log
. Compare historical logs to see if a specific module's memory usage is continuously growing, which indicates a memory leak. C. JVM Statistics Use standard JDK tools to check real-time heap usage:
Copy code
bash
# Check GC statistics (O column represents Old Gen percentage)
jstat -gcutil <fe_pid> 1000 1000

# Identify top objects in the heap (lightweight)
jmap -histo <fe_pid> | head -n 20
3. Common Causes & Solutions Identify the pattern of memory growth to apply the correct fix: Scenario A: Memory usage grows during Metadata Checkpoints Checkpointing can consume significant memory as FE loads and saves images. * Optimization: In v3.4.0+, enable checkpoint offloading to Follower nodes to reduce the Leader FE's memory load. * Optimization: In v3.3.3+, StarRocks uses "Image streaming read" to reduce the peak memory required when reading metadata images. Scenario B: High tablet count or frequent reporting If your cluster has a very large number of tablets, the tablet report queue can grow. * Fix: Upgrade to v3.5.0+, where the FE Leader actively pulls reports from BEs based on its processing capacity, preventing memory bloat in the report queue. Scenario C: Heavy Metadata or Query Planning * Query Planning: Very complex SQL queries or large 
IN
clauses can spike FE memory. In recent versions, check the 
QueryFEAllocatedMemory
metric to identify memory-heavy queries. * Catalog Metadata: If you have many external catalogs (e.g., Hive, Iceberg), metadata caching might consume heap. Check configuration related to metadata cache expiration. 4. Configuration Tuning Ensure these settings are optimized in `fe.conf`: * 
MALLOC_ARENA_MAX=1
: Add this to your environment variables or 
fe.conf
to prevent excessive virtual memory fragmentation (this helps with off-heap memory but is a standard best practice for FE). * 
proc_profile_mem_enable
: Ensure this is 
true
(default in v3.2.12+) to capture the diagnostic profiles mentioned above. If the heap usage remains high after these checks, it is recommended to collect a heap dump for deep analysis using 
jmap -dump:live,format=b,file=fe_heap.hprof <pid>
. Note: Taking a heap dump will pause the FE process; do this during a maintenance window or on a non-leader node. References • faq/fe_mem_faq.md • knowledge/trouble_shooting/fe_memory_problems.md • administration/Meta_recovery.md • administration/management/monitoring/alert.md
h
@Rocky How to set query limitation based on resource usage or how many times the query runs?
r
In StarRocks, you can limit queries based on resource usage (CPU, memory, time) or the number of concurrent executions using Resource Groups, Query Queues, and Session Variables. 1. Limit by Resource Usage (CPU, Memory, Time) Memory Limits * Per-Query Limit: Use the session variable 
query_mem_limit
to set the maximum memory a single query can use on each BE node.
Copy code
sql
    SET query_mem_limit = 8589934592; -- 8GB
* Resource Group Limit: Define a percentage of total memory for a group of queries using the 
mem_limit
property in a Resource Group.
Copy code
sql
    CREATE RESOURCE GROUP big_queries
    WITH ("mem_limit" = "0.5"); -- Limits this group to 50% of BE memory
CPU Limits * CPU Cores: In a Resource Group, you can limit the number of CPU cores available.
Copy code
sql
    CREATE RESOURCE GROUP analytical_rg
    WITH ("max_cpu_cores" = "10");
* CPU Usage Threshold: Use 
query_queue_cpu_used_permille_limit
to queue queries if the BE's total CPU usage exceeds a certain permille (e.g., 800 for 80%). Query Timeout * Limit the maximum execution time of a query to prevent long-running processes from hanging.
Copy code
sql
    SET query_timeout = 60; -- 60 seconds
────────── 2. Limit by How Many Times a Query Runs (Concurrency) StarRocks manages "how many times a query runs" primarily through concurrency control rather than a fixed daily "quota" count. Concurrency Limit (Resource Groups) You can restrict how many queries in a specific group can run at the same time. If the limit is exceeded, new queries will either fail or enter a queue.
Copy code
sql
ALTER RESOURCE GROUP dev_group 
WITH ("concurrency_limit" = 10); -- Only 10 queries from this group can run simultaneously
Global Query Queuing You can set a global threshold for concurrent queries across the entire cluster. When the limit is reached, StarRocks places new queries in a queue instead of failing them.
Copy code
sql
SET GLOBAL query_queue_concurrency_limit = 100;
User Connection Limits To limit the number of simultaneous connections a specific user can have:
Copy code
sql
ALTER USER 'jack' SET PROPERTIES ("max_user_connections" = "50");
────────── 3. Automatic "Spill to Disk" (Handling Resource Overload) If you want to prevent queries from failing when they hit memory limits, you can enable Spill to Disk. This allows memory-intensive operators (Sort, Aggregate, Join) to use disk space when memory is scarce.
Copy code
sql
SET enable_spill = true;
SET spill_mode = "auto";
Summary Table Goal | Parameter / Feature | Level Memory Limit | 
query_mem_limit
/ 
mem_limit
| Session / Resource Group CPU Limit | 
max_cpu_cores
| Resource Group Concurrency | 
concurrency_limit
| Resource Group / Query Queue Time Limit | 
query_timeout
| Session / Global Connections | 
max_user_connections
| User Property Note: For the most robust control, it is recommended to use Resource Groups to classify queries (e.g., by user or database) and apply these limits collectively. References • administration/management/resource_management/spill_to_disk.md • administration/management/BE_configuration.md • sql-reference/System_variable.md • administration/management/resource_management/resource_group.md • knowledge/usage/best-practice.md
h
@Rocky What value should be set for the following variables for a environment with limited BE resource (16 vCPU, 64GB memory)
Copy code
query_queue_concurrency_limit
query_queue_mem_used_pct_limit
query_queue_cpu_used_permille_limit
r
For an environment with limited resources like 16 vCPU and 64GB RAM, the goal of setting these variables is to prevent the BE (Backend) process from crashing due to Out-of-Memory (OOM) errors or becoming unresponsive due to 100% CPU saturation. Here are the recommended starting values and the logic behind them: 1. 
query_queue_concurrency_limit
 Recommended Value: 
16
to 
32
 * Logic: This variable limits the number of concurrent queries running on a single BE. For a 16 vCPU machine, setting this to 16 (1 query per vCPU) is conservative and ensures high predictability. If your queries are very short or have low complexity, you can increase this to 32. * Setting: 
SET GLOBAL query_queue_concurrency_limit = 16;
2. 
query_queue_mem_used_pct_limit
 Recommended Value: 
0.8
(80%) * Logic: This is the threshold for memory usage percentage on the BE. Since your BE has 64GB of RAM, StarRocks' default 
mem_limit
is usually 90% (~57GB). By setting the queue limit to 0.8 (80%), you ensure that once the BE reaches ~51GB of memory usage, new queries will wait in the queue rather than pushing the process toward a crash or triggering the expensive Linux OOM killer. * Setting: 
SET GLOBAL query_queue_mem_used_pct_limit = 0.8;
3. 
query_queue_cpu_used_permille_limit
 Recommended Value: 
800
(80%) * Logic: This is measured in "permille" (parts per thousand). A value of 800 means 80% CPU utilization. Capping query initiation at 80% CPU ensures there is always 20% headroom (roughly 3-4 vCPUs in your case) for background system tasks like Compaction, Tablet Reports, and Metadata management, which are critical for cluster stability. * Setting: 
SET GLOBAL query_queue_cpu_used_permille_limit = 800;
────────── How to apply these: You can set these globally using the following SQL command:
Copy code
sql
-- Apply to the whole cluster
SET GLOBAL query_queue_concurrency_limit = 16;
SET GLOBAL query_queue_mem_used_pct_limit = 0.8;
SET GLOBAL query_queue_cpu_used_permille_limit = 800;
Additional Tips for Limited Resources: * Check Queue Size: By default, the maximum number of queries that can wait in the queue is 1024 (
query_queue_max_queued_queries
). For a small 16 vCPU node, you might want to lower this to 100 so that users get an immediate "Too many queries" error rather than waiting indefinitely. * Timeout: Use 
query_queue_pending_timeout_second
(default 300s) to control how long a query stays in the queue before failing. On small systems, 30s to 60s is often more user-friendly. References • administration/management/resource_management/query_queues.md • faq/resource_isolation_faq.md • sql-reference/System_variable.md • administration/management/FE_configuration.md
h
@Rocky Would these configuration work for a environment with limited BE resource (16 vCPU, 64GB memory)?
Copy code
mem_limit = 90%
query_mem_limit = 8589934592 #8GB
query_timeout = 60
r
The configurations you've listed are generally safe for a StarRocks BE node with 16 vCPU and 64GB RAM, but there are some important distinctions and recommendations to keep in mind regarding where these settings are applied and how they impact your environment. 1. 
mem_limit = 90%
(BE Configuration) * Location: This belongs in your 
be.conf
file. * Analysis: For a 64GB machine, this sets the BE process limit to approximately 57.6GB. This is the standard default and is appropriate for a dedicated BE node. * Caution: If you are running other memory-intensive processes (like an FE or a database) on the same machine, you should lower this to 
60%
or 
70%
to avoid OS-level OOM (Out of Memory) kills. 2. 
query_mem_limit = 8589934592
(8GB) * Location: This is a Session Variable, not a 
be.conf
setting. You set it via SQL: 
SET GLOBAL query_mem_limit = 8589934592;
. * Analysis: This limits a single query to 8GB of RAM per BE node. * Pros: It protects your cluster from a single "runaway" query consuming all 57.6GB of available BE memory. * Cons: If you have complex queries (large joins or aggregations), 8GB might be too restrictive, causing queries to fail with "Memory limit exceeded." * Concurrency: With 57.6GB total and an 8GB limit, you could theoretically run about 7 such memory-heavy queries simultaneously. 3. 
query_timeout = 60
(60 Seconds) * Location: This is also a Session Variable: 
SET GLOBAL query_timeout = 60;
. * Analysis: The default is 300 seconds (5 minutes). * Pros: Useful for ensuring no single query hangs the system for too long. * Cons: 60 seconds is quite short for complex analytical workloads. On a 16 vCPU machine, if you are processing large datasets, some valid queries might naturally take longer than a minute. You may find yourself frequently hitting this timeout. ────────── Recommended Adjustments for your Resources: To optimize your 16 vCPU / 64GB RAM environment, I recommend also checking these settings: 1. Parallelism: Since you have 16 vCPUs, set the parallelism to utilize them effectively.
Copy code
sql
    -- Set to half or equal to the number of vCPUs
    SET GLOBAL parallel_fragment_exec_instance_num = 8;
2. Pipeline Engine: Ensure the pipeline engine is enabled (it is by default in recent versions) to better manage the 16 cores.
Copy code
sql
    SET GLOBAL enable_pipeline_engine = true;
3. Memory Management: If you find 8GB is too small for your queries, you can increase it dynamically for a specific session:
Copy code
sql
    SET query_mem_limit = 16106127360; -- Increase to 15GB for current session
Summary: Your proposed settings will work and will make the system very "safe" from heavy queries, but you might need to relax the 
query_timeout
to 300 and potentially increase 
query_mem_limit
if your queries start failing due to memory constraints. References • administration/management/resource_management/Memory_management.md • sql-reference/System_variable.md • administration/management/resource_management/Query_management.md
h
@Rocky Please check if these variables are valid in the BE config and whether this configuration fits a environment with limited BE resource (16 vCPU, 64GB memory)
Copy code
enable_spill = true
spill_mode = auto
mem_limit = 90%
report_resource_usage_interval_ms = 500
enable_query_queue_select = true
enable_query_queue_statistic = true
query_queue_concurrency_limit = 16
query_queue_mem_used_pct_limit = 0.8
query_queue_cpu_used_permille_limit = 800 #80%
query_queue_max_queued_queries = 100
query_queue_pending_timeout_second = 30
parallel_fragment_exec_instance_num = 8
enable_pipeline_engine = true
r
Hi there! I'm Rocky. Let's take a look at your configuration for that 16 vCPU, 64GB memory environment. It is important to note that most of the variables you listed are actually System Variables (set via SQL) rather than BE configuration parameters (set in 
be.conf
). Putting System Variables in 
be.conf
will have no effect. 1. Classification of Variables Variable | Where to set it | Validity/Notes 
mem_limit
| be.conf | Valid. Default is 90%. 
report_resource_usage_interval_ms
| be.conf | Valid. Default is 1000ms. 
enable_spill
| SQL 
SET GLOBAL
| System Variable. 
spill_mode
| SQL 
SET GLOBAL
| System Variable. 
enable_query_queue_select
| SQL 
SET GLOBAL
| System Variable. 
enable_query_queue_statistic
| SQL 
SET GLOBAL
| System Variable. 
query_queue_concurrency_limit
| SQL 
SET GLOBAL
| System Variable. 
query_queue_mem_used_pct_limit
| SQL 
SET GLOBAL
| System Variable. 
query_queue_cpu_used_permille_limit
| SQL 
SET GLOBAL
| System Variable. 
query_queue_max_queued_queries
| SQL 
SET GLOBAL
| System Variable. 
query_queue_pending_timeout_second
| SQL 
SET GLOBAL
| System Variable. 
parallel_fragment_exec_instance_num
| SQL 
SET GLOBAL
| System Variable. 
enable_pipeline_engine
| SQL 
SET GLOBAL
| System Variable. (True by default in v3.0+) ────────── 2. Is this configuration fit for your environment? Yes, this is a very sensible and protective configuration for a 16 vCPU / 64GB RAM environment. Here’s why: * Memory Safety (
enable_spill = true
): With only 64GB of RAM, large aggregations or joins could easily trigger Out Of Memory (OOM) errors. Enabling spilling to disk is highly recommended for stability in resource-constrained environments. * Query Queuing: Your queuing thresholds (
concurrency_limit = 16
, 
mem_used_pct_limit = 0.8
, 
cpu_used_permille_limit = 800
) are well-tuned. * Limiting concurrency to the number of vCPUs (16) prevents excessive context switching. * The 80% RAM/CPU triggers provide a safety buffer before the BE becomes unresponsive. * Reporting Frequency (
report_resource_usage_interval_ms = 500
): Reducing this from 1000ms to 500ms makes the query queue more "real-time," allowing the FE to react faster to resource spikes on the BE. * Parallelism (
parallel_fragment_exec_instance_num = 8
): Setting this to half your CPU cores is a standard conservative practice that balances performance and resource consumption. ────────── 3. Recommended Actions To apply these correctly, follow these two steps: Step A: Update 
be.conf
 Only keep these in your 
be.conf
and restart the BE:
Copy code
properties
mem_limit = 90%
report_resource_usage_interval_ms = 500
_Note: If your FE is running on the same machine as the BE, I recommend lowering 
mem_limit
to 
80%
to leave room for the FE process._ Step B: Execute SQL Commands Run these via your MySQL client to set the global variables:
Copy code
sql
SET GLOBAL enable_spill = true;
SET GLOBAL spill_mode = 'auto';
SET GLOBAL enable_query_queue_select = true;
SET GLOBAL enable_query_queue_statistic = true;
SET GLOBAL query_queue_concurrency_limit = 16;
SET GLOBAL query_queue_mem_used_pct_limit = 0.8;
SET GLOBAL query_queue_cpu_used_permille_limit = 800;
SET GLOBAL query_queue_max_queued_queries = 100;
SET GLOBAL query_queue_pending_timeout_second = 30;
SET GLOBAL parallel_fragment_exec_instance_num = 8;
-- enable_pipeline_engine is likely already true, but you can set it:
SET GLOBAL enable_pipeline_engine = true;
Let me know if you need help tuning any other parameters! References • administration/management/resource_management/query_queues.md • sql-reference/System_variable.md
h
@Rocky Would this setting be good for an environment with dedicated BE nodes (each BE has 8 vCPU, 32GB memory)
Copy code
mem_limit = 90%
report_resource_usage_interval_ms = 500

SET GLOBAL enable_spill = true;
SET GLOBAL spill_mode = 'auto';
SET GLOBAL enable_query_queue_select = true;
SET GLOBAL enable_query_queue_statistic = true;
SET GLOBAL query_queue_concurrency_limit = 8;
SET GLOBAL query_queue_mem_used_pct_limit = 0.8;
SET GLOBAL query_queue_cpu_used_permille_limit = 800;
SET GLOBAL query_queue_max_queued_queries = 100;
SET GLOBAL query_queue_pending_timeout_second = 30;
SET GLOBAL parallel_fragment_exec_instance_num = 4;
SET GLOBAL enable_pipeline_engine = true;
r
These settings are generally well-suited for a small, dedicated StarRocks environment (8 vCPU, 32GB RAM), but there are a few refinements you should consider to ensure stability and take advantage of newer features. Here is an evaluation of your configuration and some recommended adjustments: 1. Memory Configuration (
mem_limit = 90%
) * Assessment: On a 32GB node, 90% leaves approximately 3.2GB for the OS and other processes. * Recommendation: If these nodes are strictly dedicated to StarRocks BEs, 90% is fine. However, if you see the OS OOM-killing the BE process, drop this to 80% to provide a larger safety buffer for the operating system and page cache. 2. Query Queuing Settings Your queuing settings are proactive, which is excellent for a node with limited RAM. * 
query_queue_concurrency_limit = 8
: This matches your vCPU count. This is a "safe" conservative limit. StarRocks can often handle 1.5x to 2x the core count in concurrency, but starting at 8 is wise for 32GB RAM to prevent context switching and memory contention. * 
query_queue_mem_used_pct_limit = 0.8
: This triggers queuing when query memory hits ~23GB ($32GB \times 0.9 \times 0.8$). This is a solid threshold. * 
report_resource_usage_interval_ms = 500
: This makes the queue more responsive to rapid changes in load. It is a good choice for small clusters. 3. Parallelism and Pipeline Engine * 
parallel_fragment_exec_instance_num = 4
: * Recommendation: Since you have 
enable_pipeline_engine = true
, you should prefer using 
pipeline_dop
 instead. * In modern StarRocks versions (3.x+), 
pipeline_dop
defaults to 
0
, which means StarRocks will automatically set it (usually to half the CPU cores). * For your 8 vCPU node, a 
pipeline_dop
of 4 or 8 is appropriate. I recommend removing 
parallel_fragment_exec_instance_num
and letting the system auto-calculate or setting 
SET GLOBAL pipeline_dop = 4;
. * 
enable_pipeline_engine = true
: This is the correct choice and is actually the default in recent versions. 4. Spilling (
enable_spill = true
) * Assessment: Highly recommended for 32GB nodes. * Requirement: Ensure your BEs have a designated spill directory with enough disk space (configured via 
spill_local_storage_path
in 
be.conf
). Without enough disk space, spilling will fail and the query will still OOM. Suggested Refined Configuration I recommend replacing the parallelism variable and adding a per-query memory limit to prevent a single large query from hogging the entire 23GB before the queue kicks in:
Copy code
sql
-- BE Config (be.conf)
mem_limit = 90%
report_resource_usage_interval_ms = 500

-- Global Variables
SET GLOBAL enable_spill = true;
SET GLOBAL spill_mode = 'auto';
SET GLOBAL enable_query_queue_select = true;
SET GLOBAL enable_query_queue_statistic = true;
SET GLOBAL query_queue_concurrency_limit = 8;
SET GLOBAL query_queue_mem_used_pct_limit = 0.8;
SET GLOBAL query_queue_cpu_used_permille_limit = 800;
SET GLOBAL query_queue_max_queued_queries = 100;
SET GLOBAL query_queue_pending_timeout_second = 30;

-- Use pipeline_dop instead for the pipeline engine
SET GLOBAL pipeline_dop = 4; 

-- Optional: Prevent a single query from taking more than 16GB
SET GLOBAL query_mem_limit = 17179869184;
Summary for your 8vCPU/32GB environment: * Pros: Very stable; prevents OOMs through spilling and queuing. * Cons: Low concurrency limit (8) might result in queries queuing during peak hours. If you find the CPU is underutilized, you can gradually increase 
query_queue_concurrency_limit
to 12 or 16. References • administration/management/resource_management/Memory_management.md • administration/management/resource_management/query_queues.md
h
@Rocky Would this setting be good for an environment with dedicated BE nodes (each BE has 16 vCPU, 64GB memory)
Copy code
# Prod
mem_limit = 90%
report_resource_usage_interval_ms = 500

SET GLOBAL enable_spill = true;
SET GLOBAL spill_mode = 'auto';
SET GLOBAL enable_query_queue_select = true;
SET GLOBAL enable_query_queue_statistic = true;
SET GLOBAL query_queue_concurrency_limit = 16;
SET GLOBAL query_queue_mem_used_pct_limit = 0.8;
SET GLOBAL query_queue_cpu_used_permille_limit = 800;
SET GLOBAL query_queue_max_queued_queries = 100;
SET GLOBAL query_queue_pending_timeout_second = 30;
SET GLOBAL parallel_fragment_exec_instance_num = 8;
SET GLOBAL enable_pipeline_engine = true;
r
Hello! I'm Rocky. Your configuration for a 16 vCPU and 64GB RAM BE environment is a very solid "production-ready" baseline. It focuses on stability and preventing Out-Of-Memory (OOM) crashes. Here is a breakdown of your settings with some minor recommendations: 1. Memory & Reporting (BE Config) * 
mem_limit = 90%
: Perfect for dedicated BE nodes. It allows StarRocks to utilize most of the RAM while leaving a small buffer for the OS. * 
report_resource_usage_interval_ms = 500
: This is a good move. Reducing this from the default (1000ms) to 500ms makes the FE aware of resource changes faster, which makes your Query Queue much more responsive and accurate. 2. Spilling (Global Variables) * 
enable_spill = true
 and 
spill_mode = 'auto'
: Highly recommended for production. This prevents large, memory-intensive queries (like massive joins or aggregations) from failing with OOM by offloading intermediate data to disk when memory is tight. 3. Query Queue (Global Variables) These settings create a "safety net" to prevent the cluster from being overwhelmed: * 
query_queue_concurrency_limit = 16
: On a 16 vCPU machine, this allows 1 query per vCPU core. This is a conservative and safe starting point. If your queries are very short/small, you could eventually increase this to 32 or more, but 16 is great for stability. * 
query_queue_mem_used_pct_limit = 0.8
: This triggers the queue when BE memory hits 80%. Since your 
mem_limit
is 90%, this gives you a 10% safety buffer before the BE starts killing queries. * 
query_queue_cpu_used_permille_limit = 800
: Triggers the queue when CPU hits 80%. This ensures the system remains responsive for management tasks. 4. Parallelism (Global Variables) * 
parallel_fragment_exec_instance_num = 8
: This is the traditional setting (half of vCPUs). * Important Note: Since you have 
enable_pipeline_engine = true
, StarRocks (v2.3+) primarily uses 
pipeline_dop
 to control parallelism. * By default, 
pipeline_dop
is 
0
, which means StarRocks automatically sets it to half of your CPU cores (which would be 8 in your case). * If you are on a recent version (v3.x+), you don't strictly need to set 
parallel_fragment_exec_instance_num
or 
pipeline_dop
manually unless you want to override the "half-CPU" default. Rocky's Recommendations: 1. Storage for Spilling: Ensure your BEs have sufficient and fast disk space (preferably SSD/NVMe) in the 
storage_root_path
to handle the data when 
enable_spill
kicks in. 2. Version Check: If you are on StarRocks v3.1.4 or higher, the query queue is managed more efficiently by the Leader FE. Your settings are perfectly compatible with this. 3. Monitor Pending Queries: Keep an eye on 
query_queue_pending_timeout_second = 30
. If users complain about "Query waiting timeout", it means your concurrency limit (16) might be too tight for your workload, or queries are taking too long to finish. Overall, this is a very safe and well-thought-out configuration for a 16-core production environment! References • sql-reference/System_variable.md • knowledge/usage/best-practice.md • faq/resource_isolation_faq.md • best_practices/query_tuning/query_profile_operator_metrics.md • administration/management/FE_configuration.md • faq/Sql_faq.md
h
@Rocky What would be cause that there are many State=EOF in fe.audit.log?
r
In StarRocks, seeing many 
State=EOF
entries in your 
fe.audit.log
is generally normal behavior and not an indication of an error. Here is a breakdown of what this state means and why you might see many of them: 1. What "State=EOF" Means In the audit log, the 
state
field indicates the final status of a query or connection: * EOF: This stands for "End of File." In the MySQL protocol (which StarRocks uses for client connections), an EOF packet is sent to signal the successful completion of a command that returns a result set (such as 
SELECT
, 
SHOW
, or 
DESC
). * OK: This indicates the successful completion of a command that does not return a result set (such as 
INSERT
, 
UPDATE
, 
SET
, or 
USE
). * ERR: This indicates that the query failed with an error. Therefore, 
State=EOF
simply means that a query was executed successfully and returned data to the client. 2. Common Causes for "Many" EOF Entries If you notice an unusually high volume of 
EOF
entries, it is typically caused by one of the following: * Health Checks and Monitoring: Load balancers (like F5, HAProxy, or Nginx) and monitoring tools (like Prometheus, Zabbix, or Grafana) often connect to StarRocks and run frequent "heartbeat" queries (e.g., 
SELECT 1
) to verify the frontend (FE) is alive. Each of these successful checks will generate an 
EOF
entry. * High Frequency Queries: Applications or scripts that frequently run short-lived queries (e.g., 
SHOW PROCESSLIST
, 
SELECT 1
) will populate the audit log with many 
EOF
states. * Connection Module Logging: If you have enabled the 
connection
module in your FE configuration (
audit_log_modules
), StarRocks logs every login and logout. In some versions or configurations, connection termination might be logged with an 
EOF
state if the connection was closed gracefully. 3. How to Investigate To understand where these entries are coming from, check the other fields in the audit log: * 
stmt
: Look at the SQL statement being executed. If it is 
SELECT 1
or a 
SHOW
command, it is likely a health check or background monitoring. * 
clientIp
: Identify the IP address. If it belongs to a load balancer or a monitoring server, you have found the source. * 
user
: Check if a specific service account is running these queries. Summary 
State=EOF
is a success indicator for queries that return rows. Unless you are seeing a performance impact or these entries are cluttering your logs excessively, they can be ignored as part of normal system operation. If you wish to reduce log volume, you can adjust the 
audit_log_modules
or 
qe_slow_log_ms
settings in your FE configuration. References • administration/management/logs.md • administration/management/audit_loader.md • administration/management/FE_configuration.md • knowledge/usage/best-practice.md
h
@Rocky Why the leader FE consumes a lot of memory (99% usage) while the follower FEs don't (50% usage)?
r
The difference in memory usage between the Leader FE and Follower FEs is a common occurrence in StarRocks due to the specific responsibilities assigned only to the Leader node. Here are the primary reasons why your Leader FE consumes significantly more memory: 1. Metadata Checkpointing (Image Generation) Historically, the Leader FE is responsible for creating "images" (checkpoints) of the metadata. To do this, the FE essentially loads a fresh copy of the metadata into memory, replays the edit logs, and saves it to disk. * Impact: This process can cause memory usage to spike to 2x or 3x that of a Follower FE during the checkpointing period. * Optimization: Starting from v3.4.0, StarRocks supports offloading this task to Follower nodes (
image_checkpoint_offload_to_follower
), which balances memory consumption across the cluster. 2. Tablet Report Processing All Backend (BE) nodes periodically report the status of their tablets to the Leader FE only. * Impact: The Leader maintains a queue to process these reports and performs a "diff" between the reported state and the metadata state. If you have a large number of tablets, this queue and the processing logic can consume several GBs of heap memory. * Optimization: In v3.5.0, StarRocks improved this by having the Leader "pull" reports from BEs based on its own processing capacity rather than receiving them all at once, significantly reducing memory overhead. 3. Transaction and Load Management The Leader FE coordinates all transactions and data loading (Stream Load, Broker Load, etc.). It maintains the state of active transactions and coordinating metadata, which Followers do not need to hold in the same detail. 4. Metadata Catalog Caching The Leader FE is the "source of truth" and often maintains more aggressive caching for catalog metadata (especially when using External Catalogs like Hive or Iceberg) to speed up query planning. ────────── How to Troubleshoot and Mitigate If the high memory usage is causing instability (e.g., Full GC or crashes), follow these steps: 1. Check JVM Configuration: Ensure your 
fe.conf
has a proper heap size (
-Xmx
). For machines with >32GB RAM, set 
-Xmx
to roughly 80% of total memory. Also, add this environment variable to 
fe.conf
to prevent glibc memory fragmentation:
Copy code
bash
    MALLOC_ARENA_MAX=1
2. Identify the Memory Consumer: Use the built-in memory tracker to see which module is using the most space. Run this in your MySQL client:
Copy code
sql
    ADMIN SHOW FRONTEND MEMORY; -- Available in newer versions
Or check the 
fe/log/fe.audit.log
and 
fe/log/fe.log
for memory-related warnings. 3. Analyze Memory Profiles: From v3.3.6 onwards, StarRocks automatically generates memory profiles in 
fe/log/proc_profile
. You can find 
.tgz
files there containing flame graphs that show exactly which Java objects are occupying the heap. 4. Force a GC (Diagnostic only): If you want to see if the memory is "real" usage or just garbage waiting to be collected, you can manually trigger a GC:
Copy code
bash
    jcmd <fe_pid> GC.run
Recommendation: If you are on an older version (e.g., 2.x or early 3.x), upgrading to v3.3.3+ or v3.4.0 is highly recommended as it contains numerous fixes for FE metadata memory management and the "Image streaming read" optimization. References • knowledge/trouble_shooting/fe_memory_problems.md • faq/Deploy_faq.md • faq/fe_mem_faq.md • faq/operation_maintenance_faq.md • deployment/deploy_manually.md • administration/Meta_recovery.md
h
@Rocky How to read the html.tar.gz in 
fe/log/proc_profile
r
To read the 
html.tar.gz
files located in 
fe/log/proc_profile
, you have two main options: using the StarRocks Web UI (recommended) or manual extraction. Method 1: StarRocks Web UI (Recommended) StarRocks provides a built-in way to visualize these profiles as interactive Flame Graphs without manual extraction. 1. Open your browser and go to the StarRocks FE Web UI (default URL: 
http://<FE_IP>:8030
). 2. Log in (default user is 
root
with no password, unless configured otherwise). 3. Click on the proc profiles tab in the top navigation bar. 4. Select the FE tab. 5. You will see a list of collected profiles (CPU or Memory). Click the View link next to the desired entry. * The system will automatically extract the 
.tar.gz
file and render the Flame Graph directly in your browser. Method 2: Manual Extraction If you want to view the file offline or on another machine, you can extract it manually: 1. Locate the file in the terminal:
Copy code
bash
    cd /path/to/starrocks/fe/log/proc_profile
2. Extract the archive:
Copy code
bash
    tar -zxvf cpu-profile-20231027-100000.html.tar.gz
(Note: Replace the filename with your actual filename.) 3. Open the HTML file: The extraction will produce an 
.html
file. You can open this file in any modern web browser (Chrome, Firefox, etc.) to view the interactive Flame Graph. Context * What are these files? These are process-level profiles generated by 
AsyncProfiler
(for FE). They capture Java stack traces to help diagnose CPU hotspots or memory allocation issues. * Configuration: This feature is controlled by FE parameters like 
proc_profile_cpu_enable
and 
proc_profile_mem_enable
(available since v3.2.12). * Retention: By default, these files are kept for 1 day or until the directory reaches 2GB, as controlled by 
proc_profile_file_retained_days
and 
proc_profile_file_retained_size_bytes
. References • administration/management/FE_configuration.md • administration/management/proc_profile.md • administration/management/logs.md
h
@Rocky Why this is no proc profiles tab in StarRocks FE Web UI?
r
The "proc profiles" tab is a feature of the Process Profile (Proc Profile) mechanism, which was introduced in StarRocks v3.2.12 and v3.3.3. If you do not see this tab in your Frontend (FE) Web UI, it is likely due to one of the following reasons: 1. Version Compatibility The "proc profiles" tab is only available in: * v3.2.12 and later versions in the 3.2 branch. * v3.3.3 and later versions in the 3.3 branch. * v3.4.0 and all subsequent major releases. If you are on an older version (e.g., v2.5.x, v3.0.x, or early 3.1.x/3.2.x versions), this tab does not exist. You would need to upgrade to a supported version to use this built-in flame graph visualization tool. 2. Configuration Settings Even if you are on a supported version, the feature must be enabled in your 
fe.conf
. By default, these are 
true
, but you should verify them: * `proc_profile_cpu_enable`: Enables automatic CPU profiling for FE. * `proc_profile_mem_enable`: Enables automatic memory allocation profiling for FE. If these are set to 
false
, the collection won't happen, though the tab should generally still be visible in supported versions. 3. Difference between "proc profiles" and "queries" Please ensure you are looking for the correct tool: * queries tab: Used for Query Profiles (analyzing specific SQL execution performance). * proc profiles tab: Used for Process Profiles (system-level diagnosis like CPU/Memory flame graphs for the FE/BE processes themselves). How to use it (if on a supported version): 1. Access the Web UI at 
http://<fe_ip>:<fe_http_port>
(default port is 
8030
). 2. Click the proc profiles tab in the top navigation bar. 3. You can then switch between the FE tab and various BE node tabs to view collected flame graphs. Note for BE Profiles: While FE profiles are collected automatically by an internal daemon, BE profiles often require running the 
collect_be_profile.sh
script (found in the 
bin/
directory) to trigger or schedule the collection of data before they appear in the Web UI. References • administration/management/proc_profile.md • administration/management/FE_configuration.md • best_practices/query_tuning/query_profile_text_based_analysis.md
h
@Rocky Why this is no proc profiles tab in StarRocks FE Web UI? Starrocks version is 3.5.8 and there are many html.tar.gz in 
fe/log/proc_profile
r
The Proc Profiles (Process Profile) feature was introduced in StarRocks version 3.2.12. Since you are on version 3.5.8, the feature is available and, based on the presence of 
html.tar.gz
files in your 
fe/log/proc_profile
directory, the background collection is working correctly. If you cannot see the proc profiles tab in the FE Web UI, it is likely due to one of the following reasons: 1. You are not accessing the Leader FE Many management and diagnostic tabs in the StarRocks Web UI are only visible or fully functional when accessing the Leader FE. * Action: Check which FE is the leader using 
SHOW FRONTENDS;
in your SQL client. * Action: Ensure you are browsing the Web UI of that specific Leader FE (usually at 
http://<leader_fe_ip>:8030
). 2. Permissions/Login Status The proc profiles tab provides sensitive system-level information and is typically restricted to users with administrative privileges. * Action: Ensure you are logged into the Web UI as the 
root
or 
admin
user. 3. Check for the Tab Location In recent versions, the Web UI layout may have changed. * Look for a top-level tab labeled proc profiles. * If you don't see it, check if it's nested under a System or Nodes menu. * Direct URL: You can attempt to access the page directly by appending 
/proc_profile
to your FE Web UI URL, for example: 
http://<fe_ip>:8030/proc_profile
. 4. Browser Cache or UI Version Sometimes the browser may cache an older version of the Web UI. * Action: Try a hard refresh (
Ctrl + F5
or 
Cmd + Shift + R
) or open the Web UI in an Incognito/Private window. Why the files exist but the tab might be missing The 
html.tar.gz
files are generated by a background daemon (
ProcProfileCollector
). The configuration parameters 
proc_profile_cpu_enable
and 
proc_profile_mem_enable
control this collection and are 
true
by default. The existence of these files proves the "backend" of this feature is active; the "frontend" (the Web UI tab) simply needs to be accessed via the correct node and user. Summary of FE configuration related to this (in 
fe.conf
): * `proc_profile_cpu_enable`: Defaults to 
true
. * `proc_profile_mem_enable`: Defaults to 
true
. * `proc_profile_collect_time_s`: Duration of sampling (default 120s). If you are on the Leader FE and logged in as admin but still cannot see the tab, verify if there are any errors in the 
fe.log
related to the 
ProcProfileCollector
. References • administration/management/proc_profile.md • administration/management/FE_configuration.md
h
@Rocky Why this is no proc profiles tab in StarRocks FE Web UI? Starrocks version is 3.5.8 and there are many html.tar.gz in 
fe/log/proc_profile
It is the leader FE Web UI that is being accessed, using the admin user
r
The "proc profiles" tab was introduced in StarRocks v3.2.12 to provide a visual interface for process-level performance analysis (Flame Graphs). Since you are on version 3.5.8 and see 
html.tar.gz
files in 
fe/log/proc_profile
, it confirms that the background collection is working correctly. If the tab is missing from the Web UI while you are accessing the Leader FE as an admin user, please check the following: 1. Verify Leader Status Even if you are accessing a specific IP, ensure it is the current leader. Only the Leader FE provides the full management interface for profiles. * Run 
SHOW FRONTENDS;
in your SQL client. * Check the 
IsMaster
column to confirm the IP you are accessing is indeed the Leader. 2. UI Navigation In version 3.5.x, the Web UI has been updated. The "proc profiles" tab is typically located at the top-level navigation bar. If you don't see it: * Try accessing the path directly: 
http://<FE_IP>:<FE_HTTP_PORT>/#/proc_profile
* Ensure you are using the "New UI" (StarRocks 3.0+ defaults to a modern dashboard). If you are using an older bookmark or a specific path that leads to the "Old UI" (e.g., 
/home
), the tab may not be present. 3. Check FE Configuration While the collection parameters (
proc_profile_cpu_enable
, 
proc_profile_mem_enable
) are 
true
by default and clearly working (since files exist), verify if there are any restrictions. * Ensure that no proxy or firewall is stripping out specific UI components or blocking the port (default 8030). * Check if the 
admin
user has the necessary permissions. Although 
admin
should have everything, ensure you haven't restricted the 
NODE_PRIV
or 
SYSTEM_PRIV
. 4. Browser Cache Since you recently upgraded or are using a newer version, the browser might be caching older Frontend assets. * Try to Hard Refresh (Ctrl + F5 or Cmd + Shift + R) or open the Web UI in an Incognito/Private window to ensure the latest UI components are loaded. 5. Backend (BE) Profiles Note that the "proc profiles" tab also allows viewing BE profiles. However, BE profiles are not collected automatically like FE profiles. * To see BE entries in that tab, you must run the collection script on the BE nodes:
Copy code
bash
    # Example: Collect a 30-second CPU profile for a BE
    ./bin/collect_be_profile.sh --profiling-type cpu --duration 30
* Once collected, the BE profiles will appear in the same proc profiles tab under the BE sub-tab. If the tab is still missing after a hard refresh on the confirmed Leader FE, check the FE log (
fe.log
) for any errors related to 
ProcProfileCollector
or 
HttpServer
that might indicate the UI component failed to initialize. References • administration/management/FE_configuration.md • administration/management/proc_profile.md
h
@Rocky Why this is no proc profiles tab in StarRocks FE Web UI? Starrocks version is 3.5.8 and there are many html.tar.gz in 
fe/log/proc_profile
It is the leader FE Web UI that is being accessed, using the admin user. When trying to access the path 
http://<FE_IP>:<FE_HTTP_PORT>/proc_profile
, this message showed
Copy code
this is 404 page.
r
h
@Rocky What could have been the reasons for this error?
Copy code
2026-03-23 11:26:28.785Z ERROR (stats-cache-refresher-3|149) [ColumnBasicStatsCacheLoader.lambda$asyncLoadAll$1():123] com.starrocks.sql.analyzer.SemanticException: Getting analyzing error. Detail message: Statistics query fail | Error Message [INTERNAL_ERROR]
r
This error occurs when the StarRocks Frontend (FE) fails to load table/column statistics into its metadata cache. The optimizer requires these statistics to generate efficient execution plans. When the cache loader (
ColumnBasicStatsCacheLoader
) attempts to fetch this data from the internal 
_statistics_
database and the query fails, it throws a 
SemanticException
. The 
INTERNAL_ERROR
suggests that the internal query executed by the FE encountered a failure at the execution level. Common Reasons for this Error 1. Unhealthy 
_statistics_
Database: StarRocks stores statistics in internal tables within the 
_statistics_
database. If the tablets for these tables (specifically 
column_statistics
) are unhealthy or have insufficient replicas, the internal fetch query will fail. * Check health: Run 
SHOW TABLET FROM _statistics_.column_statistics;
or 
ADMIN SHOW REPLICA STATUS FROM _statistics_.column_statistics;
to see if any tablets are offline. 2. Internal Query Timeout: If your cluster has a massive number of columns and the 
column_statistics
table has grown very large, the internal query used to refresh the cache might time out. 3. Backend (BE) Issues: Since it is an 
INTERNAL_ERROR
, the failure often happens on the BE side during query execution. * Check the 
<http://be.INFO|be.INFO>
or 
be.WARNING
logs on your backends around the same timestamp (
2026-03-23 11:26:28
) for errors like 
process query failed
or memory-related issues. 4. Metadata Inconsistency: In rare cases, especially after upgrades or improper manual deletions in the 
_statistics_
database, the metadata in the stats tables might become inconsistent with the current cluster state. 5. Concurrent Analyze Jobs: Heavy concurrent 
ANALYZE
tasks or massive data loading can sometimes cause contention or resource exhaustion that impacts the stability of internal statistics queries. Recommended Troubleshooting Steps 1. Check 
fe.internal.log
: StarRocks maintains a specific log for internal operations. Check 
fe.internal.log
for the specific internal SQL statement that failed and the detailed stack trace. This will often show the exact SQL being run (e.g., 
SELECT ... FROM _statistics_.column_statistics
). 2. Verify Statistics Table Health: Run the following to ensure the statistics database is functional:
Copy code
sql
    -- Check if the table is accessible
    SELECT count(*) FROM _statistics_.column_statistics;

    -- Check replica status
    ADMIN SHOW REPLICA STATUS FROM _statistics_.column_statistics WHERE State != 'OK';
3. Manual Statistics Refresh: If the error is isolated to a specific table, try manually re-collecting statistics to fix potential data corruption in the cache:
Copy code
sql
    ANALYZE TABLE <your_db>.<your_table>;
4. Cleanup Old Statistics: If the 
_statistics_
tables have grown excessively large, you can try to truncate or delete old entries, although it is usually safer to let the system manage them or use 
DROP STATS
. 5. Increase FE Memory: If the 
fe.log
shows 
OutOfMemoryError
or GC issues, the statistics cache might be too large for the current FE JVM heap size. You may need to increase 
-Xmx
in 
fe.conf
. References • loading/loading_introduction/troubleshooting_loading.md • using_starrocks/caching/block_cache.md • data_source/data_cache_observe.md • administration/management/monitoring/metrics.md • faq/operation_maintenance_faq.md • faq/loading/Loading_faq.md
h
@Rocky How to persist global variables after restart? These global variables can only be set through the mysql interface and cannot be set in the configuration files
r
In StarRocks, there is an important distinction between System Variables and Configuration Items regarding how they persist after a restart. 1. System Variables (SET GLOBAL) If the variables you are referring to are System Variables (the ones you see when running 
SHOW VARIABLES
), using the 
SET GLOBAL
command will automatically persist them after a restart. * Persistence Mechanism: StarRocks stores global system variables in the Frontend (FE) metadata (using BDBJE/Edit Logs). When an FE restarts, it reloads these variables from its persistent metadata store. * Command:
Copy code
sql
    SET GLOBAL variable_name = value;
* Note: 
SET GLOBAL
only affects new sessions. To apply the change to existing long-lived connections without restarting them, you can use the 
REFRESH CONNECTIONS
command (available in newer versions):
Copy code
sql
    REFRESH CONNECTIONS;
2. FE/BE Configuration Items (ADMIN SET CONFIG) If you are using the 
ADMIN SET FRONTEND CONFIG
or 
ADMIN SET BACKEND CONFIG
commands to change cluster configurations (the ones you see in 
ADMIN SHOW FRONTEND CONFIG
or 
SHOW BACKEND CONFIG
), these do NOT persist across restarts by default. * Behavior: These commands are used for dynamic configuration changes at runtime. However, when the service restarts, it will read the values from the 
fe.conf
or 
be.conf
files. * How to Persist: Even if you feel they "cannot" be set in configuration files, the standard way to persist these is to add the corresponding key-value pair directly into the 
fe.conf
or 
be.conf
file on each node. * Example for fe.conf:
Copy code
properties
    # Add this line to fe.conf to make it permanent
    disable_balance = true
Summary Table Type | Set Command | Persistent After Restart? System Variables | 
SET GLOBAL ...
| Yes (Stored in FE Metadata) FE Config | 
ADMIN SET FRONTEND CONFIG ...
| No (Must be added to 
fe.conf
) BE Config | 
ADMIN SET BACKEND CONFIG ...
| No (Must be added to 
be.conf
) Rocky's Tip: If you're unsure which one you're dealing with, check 
SHOW VARIABLES
. If your variable is listed there, simply running 
SET GLOBAL <var> = <val>;
is enough to keep it through a restart! References • sql-reference/System_variable.md • sql-reference/sql-statements/cluster-management/config_vars/SET.md • faq/operation_maintenance_faq.md • sql-reference/sql-statements/cluster-management/config_vars/ADMIN_SET_CONFIG.md • deployment/post_deployment_setup.md
h
@Rocky Can System Variables be set in the fe or be config instead of running SET GLOBAL?
r
Hi! I'm Rocky. To answer your question: In StarRocks, System Variables and FE/BE Configuration Parameters are two distinct sets of settings, and they are managed differently. Short Answer: No You generally cannot set standard System Variables (like 
query_timeout
, 
exec_mem_limit
, or 
parallel_fragment_exec_instance_num
) directly in the 
fe.conf
or 
be.conf
files. Why? 1. Persistence: System variables set with 
SET GLOBAL
are automatically persistent. They are stored in the Frontends' metadata (BDBJE), meaning they survive cluster restarts. Because they are already permanent once set globally, there is no need to define them in a configuration file. 2. Scope: * Configuration Parameters (`fe.conf`/`be.conf`): These control the process-level behavior (e.g., port numbers, log paths, JVM heap size). * System Variables (
SET
): These control the SQL execution engine's behavior and can be tuned at the Global, Session, or even Query level (via 
SET_VAR
hints). How to set "Defaults" permanently If your goal is to ensure a specific variable is always set for all users without running 
SET GLOBAL
every time the cluster is created, here are your options: 1. Run 
SET GLOBAL
once:
Copy code
sql
    SET GLOBAL query_timeout = 300;
This change is recorded in the FE metadata and will remain in effect even after you restart the entire cluster. 2. Set Properties for Specific Users: Starting from v3.3.3, you can set default session variables for specific users using `ALTER USER`:
Copy code
sql
    ALTER USER 'username' SET PROPERTIES ('session.query_timeout' = '600');
3. Exception (FE Configuration): There are a few rare cases where an FE configuration parameter in 
fe.conf
might influence a system variable (for example, 
mysql_server_version
in 
fe.conf
sets the value of the global 
version
variable), but these are specific exceptions and not a general rule for all variables. If you have a specific variable in mind that you're trying to configure, let me know and I can double-check if there's a specialized config for it! References • sql-reference/System_variable.md • administration/management/FE_configuration.md • sql-reference/sql-statements/cluster-management/config_vars/SET.md • administration/management/BE_configuration.md
h
@Rocky We got "too many versions" error. How can we tweak the setting for an environment with dedicated BE nodes (each BE has 16 vCPU, 64GB memory) to deal with the error?
Copy code
cumulative_compaction_num_threads_per_disk = 4
base_compaction_num_threads_per_disk = 2
cumulative_compaction_check_interval_seconds = 2
update_compaction_num_threads_per_disk = 4
update_compaction_per_tablet_min_interval_seconds = 30
r
The "too many versions" error indicates that the number of data versions in a tablet has exceeded the safety limit (default is 1000), which happens when data loading is too frequent and the compaction process cannot keep up with merging these small versions into larger ones. For a BE node with 16 vCPU and 64GB RAM, your current settings are already quite aggressive. However, you can optimize them further or adjust the threshold to better handle the load. 1. Increase the Version Threshold If your query performance is still acceptable, the most direct way to stop the error is to increase the maximum allowed versions. • 
tablet_max_versions
: Increase this from the default 
1000
to 
2000
or even 
5000
.
Copy code
properties
   # be.conf
   tablet_max_versions = 2000
Note: Higher values may slightly increase query latency for some tables, but it prevents the "too many versions" write failure. 2. Refine Compaction Threads for 16 vCPU Your current settings use 
_per_disk
suffixes. If your BE has multiple disks (e.g., 4 disks), a setting of 
4
threads per disk would result in 16 threads for cumulative compaction alone, which might saturate your 16 vCPUs and cause resource contention. For a 16 vCPU environment, a better rule of thumb is to keep total compaction threads around 25%–50% of total CPU cores. • If you have many disks, reduce the per-disk count:
Copy code
properties
  cumulative_compaction_num_threads_per_disk = 2
  base_compaction_num_threads_per_disk = 1
• If you are on StarRocks 3.1+, it is recommended to use the Event-based Compaction Framework (enabled by default) and tune the global concurrency:
Copy code
properties
  # Total threads for all compaction types
  compact_threads = 4
  # Maximum concurrency
  max_compaction_concurrency = 8
3. Primary Key Table Optimizations Since you mentioned 
update_compaction_num_threads_per_disk
, it implies you are using Primary Key tables. These tables use a different compaction mechanism. • 
update_compaction_num_threads_per_disk
: 4 is generally fine, but ensure the total doesn't exceed vCPU count. • 
update_compaction_per_tablet_min_interval_seconds
: You have it at 
30
. If you have a very high frequency of small updates to the same tablets, you can try decreasing this to 
10
or 
20
to trigger compaction more often, but monitor CPU usage. 4. Adjust Compaction Strategy • 
cumulative_compaction_check_interval_seconds = 2
: This is already very low (aggressive). • 
max_cumulative_compaction_num_singleton_deltas
: The default is 
1000
. You can try reducing this (e.g., to 
100
or 
200
) to force the system to merge smaller batches of versions more frequently, which helps reduce the version count faster. 5. Best Practice: Batch Your Loads The "too many versions" error is often a symptom of "small file syndrome." • Increase Batch Size: Instead of many small loads every second, try to batch data so that you load every 5–10 seconds or wait until you have a larger MB/row count. • Stream Load: If using Stream Load, ensure you aren't sending thousands of tiny requests per minute. Summary of Recommended 
be.conf
for your Specs:
Copy code
properties
# Increase threshold to prevent immediate failure
tablet_max_versions = 2000

# For 16 vCPU, avoid over-saturating CPU if multiple disks exist
cumulative_compaction_num_threads_per_disk = 2
base_compaction_num_threads_per_disk = 1
update_compaction_num_threads_per_disk = 2

# For Primary Key tables: trigger more frequent merges if needed
update_compaction_per_tablet_min_interval_seconds = 20

# Memory safety for 64GB RAM
compaction_max_memory_limit_percent = 30
Monitoring Tip: Use 
SHOW PROC '/compactions';
to check the compaction score. If the 
MaxScore
is consistently high, it means compaction is still lagging behind your ingestion rate. References • administration/management/BE_configuration.md
h
@Rocky We got "too many versions" error. How can we tweak the setting for an environment with dedicated BE nodes (each BE has 16 vCPU, 64GB memory) to deal with the error? There is only one disk per BE and we use primary key tables. Besides, the CPU usage is already high at 96% when the ETL/writes runs.
Copy code
cumulative_compaction_num_threads_per_disk = 4
base_compaction_num_threads_per_disk = 2
cumulative_compaction_check_interval_seconds = 2
update_compaction_num_threads_per_disk = 4
update_compaction_per_tablet_min_interval_seconds = 30
r
The "too many versions" error indicates that your ingestion rate is producing data versions (rowsets) faster than the background compaction process can merge them. Since your CPU is already at 96% on a 16-vCPU machine, simply increasing thread counts may further saturate your CPU and degrade performance. For Primary Key (PK) tables, the compaction behavior is governed by specific "update" configurations. Here is a plan to address the error while managing high CPU usage: 1. Immediate Relief: Increase Version Limit To stop the immediate ingestion failures, increase the maximum allowed versions. This gives the system more "breathing room" to catch up. * Parameter: 
tablet_max_versions
(Default: 1000) * Action: Increase to 
2000
or 
3000
. * Where: 
be.conf
(requires restart) or via 
ADMIN SET CONFIG
. 2. Improve Compaction Efficiency (Reduce CPU Overhead) Your current 
update_compaction_per_tablet_min_interval_seconds = 30
is very aggressive. This forces StarRocks to compact a tablet every 30 seconds if new data arrives. This creates many small compaction tasks, which have high CPU overhead relative to the amount of data merged. * Action: Increase 
update_compaction_per_tablet_min_interval_seconds
to 60 or 120. * Effect: This allows more rowsets to accumulate before a compaction triggers. Merging 50 rowsets in one task is significantly more CPU-efficient than running 5 tasks to merge 10 rowsets each. 3. Rebalance Resource Allocation On a 16-vCPU machine with 96% load, your current configuration allocates 10 threads (4 update + 4 cumulative + 2 base) to compaction. This is quite high for a 16-core system under heavy ETL. Since you are using Primary Key tables, they primarily use the 
update_compaction
threads. The 
cumulative
and 
base
threads are largely used for other table models (Duplicate/Unique/Aggregate). * Action: * Keep 
update_compaction_num_threads_per_disk = 4
(this is the 25% recommendation for your 16 cores). * Reduce 
cumulative_compaction_num_threads_per_disk
to 1 or 2. * Reduce 
base_compaction_num_threads_per_disk
to 1. * Increase 
cumulative_compaction_check_interval_seconds
to 5 or 10 (2 seconds is very frequent and adds unnecessary polling overhead). 4. Enable Size-Tiered Compaction Ensure that the size-tiered strategy is enabled, as it is much more efficient for PK tables in high-write scenarios. * Parameter: 
enable_pk_size_tiered_compaction_strategy = true
(Default in newer versions, check your version). 5. Upstream Batching (The Root Cause) The "too many versions" error is almost always caused by frequent, small writes. If you are using Stream Load or Routine Load: * Routine Load: Increase 
max_routine_load_batch_size
(e.g., to 512MB or 1GB) and 
routine_load_task_consume_second
(e.g., to 15-30s). * Stream Load: Ensure your client is batching data so that you aren't sending a commit every few seconds. Aim for commits every 10–30 seconds if possible. Recommended 
be.conf
Summary:
Copy code
properties
# Increase threshold to prevent ingestion failure
tablet_max_versions = 2000

# Efficiency: Allow more data to accumulate to make each CPU cycle more productive
update_compaction_per_tablet_min_interval_seconds = 120

# Resource Rebalancing: Focus threads on PK compaction, reduce others to save CPU
update_compaction_num_threads_per_disk = 4
cumulative_compaction_num_threads_per_disk = 1
base_compaction_num_threads_per_disk = 1

# Reduce polling frequency
cumulative_compaction_check_interval_seconds = 5
Note: If CPU remains at 96% after these changes, you may need to scale horizontally (add more BE nodes) or vertically, as the current workload is at the physical limit of 16 vCPUs. References • best_practices/primarykey_table.md • table_design/table_types/primary_key_table.md
h
@Rocky What does this error mean?
Copy code
ERROR 1064 (HY000): Backend node not found. Check if any backend node is down.backend: [10.10.10.10 alive: true inBlacklist: true
r
This error message indicates that while the Backend (BE) node at 
10.10.10.10
is technically alive (the heartbeat from the Frontend is successful), it has been placed on the Backend Blacklist. Because the node is blacklisted, the Frontend (FE) excludes it from query execution to avoid potential failures, which leads to the "Backend node not found" error when the FE tries to schedule the query. Why is the node blacklisted? There are two ways a node ends up in the blacklist: 1. Automatic Blacklisting (
AUTO
): StarRocks automatically blacklists a BE if the FE detects frequent connection failures or query timeouts to that specific node. This is a self-healing mechanism to prevent a "gray failure" (a node that is up but not performing correctly) from impacting the whole cluster. 2. Manual Blacklisting (
MANUAL
): An administrator manually added the node to the blacklist using the 
ADD BACKEND BLACKLIST
command. How to troubleshoot and fix 1. Check the Blacklist Status Run the following command to see why the node was blacklisted and whether it was an automatic or manual action:
Copy code
sql
SHOW BACKEND BLACKLIST;
* AddBlackListType: 
AUTO
means StarRocks detected issues; 
MANUAL
means it was added by a user. * LostConnectionNumberInPeriod: Shows how many connection failures were detected. 2. If the type is 
MANUAL
 You must manually remove it from the blacklist to allow it to process queries again:
Copy code
sql
-- Get the BackendId from 'SHOW BACKENDS'
DELETE BACKEND BLACKLIST <backend_id>;
3. If the type is 
AUTO
 StarRocks will automatically remove the node from the blacklist once the connection becomes stable (it checks connectivity periodically). If it stays in the blacklist: * Check Network: Ensure there are no firewall rules or intermittent network issues between the FE and BE. * Check BE Health: Look at the BE's 
<http://be.INFO|be.INFO>
logs for crashes, OOM (Out of Memory) issues, or long GC pauses. * FE Configurations: The behavior is controlled by these FE parameters: * `black_host_connect_failures_within_time`: Max allowed failures before blacklisting (default is 5). * `black_host_history_sec`: How long to track failures (default is 3600s). Summary The FE is "protecting" your cluster by not sending work to a node it perceives as unreliable. Verify the network stability between your FE and the BE at 
10.10.10.10
. References • administration/management/BE_blacklist.md • sql-reference/sql-statements/cluster-management/nodes_processes/SHOW_BACKEND_BLACKLIST.md • sql-reference/sql-statements/cluster-management/nodes_processes/ADD_BACKEND_BLACKLIST.md • sql-reference/sql-statements/cluster-management/nodes_processes/DELETE_BACKEND_BLACKLIST.md • administration/management/FE_configuration.md
h
@Rocky What does this error from FE mean?
Copy code
Failed to compute statement digest
r
The error 
Failed to compute statement digest
 is related to the SQL Digest feature introduced in StarRocks v3.3.6. What it means A SQL Digest is a fingerprint (hash) generated for SQL statements after removing specific parameters (like constants, 
IN
list values, or 
LIMIT
counts). This allows StarRocks to group similar queries together in the audit logs (
fe.audit.log
) for performance analysis. When you see this error, it means the Front-End (FE) failed to normalize the current SQL statement into its "parameterless" form to generate that hash. Potential Causes 1. Unsupported SQL Syntax: The normalization engine might encounter a complex or non-standard SQL construct that it doesn't yet know how to strip of parameters. 2. Configuration: This feature is controlled by the FE configuration 
enable_sql_digest
. If it is set to 
true
(it is often disabled by default or requires manual enablement in some versions), the FE will attempt to compute this for every query. 3. Large/Deeply Nested Queries: Very large or deeply nested queries might occasionally cause issues during the normalization phase. Impact In most cases, this error does not stop the query from executing. It typically results in: * An error message logged in 
fe.log
or 
fe.audit.log
. * The 
Digest
field in your audit logs being empty or marked as failed for that specific query. How to Resolve or Troubleshoot * Check FE Logs: Look at 
fe.log
for a stack trace following this error message. It will often reveal the specific SQL part that caused the failure. * Verify Version: Ensure you are on v3.3.6 or later if you intend to use this feature. * Disable the Feature: If the error is frequent and you do not need SQL Digesting for your audit analysis, you can disable it dynamically:
Copy code
sql
    ADMIN SET FRONTEND CONFIG ('enable_sql_digest' = 'false');
* Check for Fixes: If you are using a very early 3.3.x version, consider upgrading to the latest patch release (e.g., 3.3.8+), as improvements to the SQL normalizer are frequently added. If you can provide the specific SQL query that triggered this, I can help identify if it's a known unsupported pattern! References • release_notes/release-2.3.md • release_notes/release-2.2.md • release_notes/release-3.1.md • release_notes/release-2.0.md • release_notes/release-2.1.md • release_notes/release-2.5.md
h
@Rocky What does this error mean?
Copy code
close channel failed. channel_name=NodeChannel[10000], load_info=load_id=eea3628c-2cdf-11f1-bcf4-111111111111, txn_id: 11111111, parallel=1, compress_type=2, error_msg=10.10.10.10: PTabletWriterAddSegmentRequest tablet_id 1111111 not exists
r
The error message 
PTabletWriterAddSegmentRequest tablet_id 1111111 not exists
typically occurs during a data loading process (such as Stream Load, Broker Load, or Routine Load). It indicates that the Backend (BE) node receiving the data segment was asked to write to a tablet ID that it does not currently have in its local metadata or memory. Common Causes 1. Tablet Migration or Rebalancing: While the load job was running, the Frontend (FE) may have decided to move the tablet from one BE to another (e.g., due to disk balance or node decommissioning). The load task, still using the old location info, fails when it tries to send data to the original BE. 2. Schema Change or Truncation: If you performed an 
ALTER TABLE
(like adding columns) or a 
TRUNCATE TABLE
command while a load was in progress, the old tablets might have been deleted or replaced by new ones. 3. Replica Failure: If the FE marked a replica on that specific BE as "bad" or "stale" (due to a previous disk error or version lag), it might have been scheduled for deletion or ignored, leading to this error if a load task still tried to reach it. 4. BE Restart or Metadata Inconsistency: If a BE was recently restarted, there might be a brief window where the tablet metadata is being re-synced from the FE, or in rare cases, a discrepancy between the FE's view of tablet locations and the actual BE state. How to Troubleshoot 1. Check Tablet Status: Run the following SQL to see where the tablet is currently located and its status:
Copy code
sql
    SHOW TABLET 1111111;
* Check the 
BackendId
and 
State
columns. * Verify if the 
IsSync
column is 
true
. 2. Check for Concurrent Operations: Check if any 
ALTER TABLE
, 
DROP
, or 
TRUNCATE
operations were performed on that table at the time of the error. 3. Check FE/BE Logs: * FE logs (
fe.log
): Look for messages related to "tablet balancer," "migrate," or "clone" for that tablet ID. * BE logs (
<http://be.INFO|be.INFO>
): Look for why the tablet was removed or if there were any disk/IO errors. 4. Retry the Load: Usually, this is a transient issue. If the load failed because of a migration, retrying the load will allow the FE to provide the updated, correct tablet locations to the load process. 5. Verify Replica Integrity: If the error persists, you can check if the table has enough healthy replicas:
Copy code
sql
    ADMIN SHOW REPLICA STATUS FROM <table_name>;
If this happens frequently during high-load periods, it might suggest that your cluster is frequently rebalancing data, which can be tuned by adjusting parameters like 
tablet_sched_concurrency
in the FE configuration. References • faq/loading/Loading_faq.md • sql-reference/sql-statements/table_bucket_part_index/SHOW_TABLET.md • administration/management/BE_configuration.md • faq/Sql_faq.md • faq/operation_maintenance_faq.md • table_design/table_types/primary_key_table.md • administration/management/monitoring/metrics.md • faq/loading/Broker_load_faq.md
h
@Rocky Got this error during streamload. The tablet id does exist after checking with show tablet 1111111. IsSync is true. IsBad is false. IsSetBadForce is false.
Copy code
PTabletWriterAddSegmentRequest tablet_id 1111111 not exists
r
The error 
PTabletWriterAddSegmentRequest tablet_id 1111111 not exists
during a Stream Load typically indicates that while the tablet itself exists in the cluster (as you've verified with 
SHOW TABLET
), the in-memory TabletWriter object on the Backend (BE) has been destroyed or was never successfully initialized for this specific load transaction. In StarRocks, when a load starts, the BE creates a 
TabletWriter
to manage the data being written for a specific tablet. If this writer is missing when the BE tries to add a segment, it throws this error. Common Causes and Solutions 1. Load Timeout (Most Likely) If the Stream Load takes longer than the configured timeout, the FE or BE may abort the transaction and clean up the writers. * BE Side: The parameter 
streaming_load_rpc_max_alive_time_sec
(default 1200s) in 
be.conf
controls how long a writer can stay alive. If your load is very large, it might exceed this. * FE Side: The default timeout for a Stream Load job is 600 seconds. You can increase this by passing a 
timeout
header in your Stream Load request:
Copy code
bash
    curl --location-trusted -u user:passwd \
         -H "timeout:3600" \
         -T data.csv <http://fe_host:http_port/api/db/table/_stream_load>
2. BE Node Restart or Crash If the BE node where tablet 
1111111
resides restarted during the load, all active 
TabletWriter
objects in memory are lost. * Check: Check the BE uptime or the 
<http://be.INFO|be.INFO>
logs for any crash signals or "Starting StarRocks BE" messages around the time of the error. 3. High Memory Pressure If the BE is under extreme memory pressure, it might fail to allocate resources or, in some cases, the process might be struggling to keep up with RPC requests, leading to internal timeouts that trigger writer cleanup. * Check: Look for "Memory limit exceeded" or "OOM" messages in the 
<http://be.INFO|be.INFO>
logs. 4. Concurrent Schema Change If a schema change (like adding a column or changing a distribution) was performed on the table while the load was active, the tablet version might have changed, causing the current load's writer to become invalid. Recommended Troubleshooting Steps 1. Check Stream Load Response: Look at the JSON response of your failed Stream Load. It often contains a more descriptive 
Message
field. 2. Inspect BE Logs: Search the 
<http://be.INFO|be.INFO>
logs on the BE node for the specific tablet ID 
1111111
. Look for entries related to 
TabletWriterMgr
or 
cancel
to see why the writer was removed. 3. Adjust Timeouts: If you are loading a large amount of data (several GBs), ensure you increase the timeout: * In the 
curl
command, add 
-H "timeout:3600"
. * If necessary, increase 
streaming_load_rpc_max_alive_time_sec
in 
be.conf
and restart the BE. 4. Verify Tablet Distribution: Run 
SHOW TABLET 1111111
and check which BEs host the replicas. If only one BE is reporting the error, focus on the logs of that specific BE. Summary: The error doesn't mean the tablet is gone from the system; it means the "writing session" for that tablet on the BE was closed prematurely. Usually, increasing the load timeout resolves this. References • administration/management/BE_configuration.md • faq/loading/Loading_faq.md • loading/StreamLoad.md • administration/management/FE_configuration.md
h
@Rocky Is this load type stream load or sql insert?
Copy code
LoadJobSourceType: INSERT_STREAMING
r
The 
LoadJobSourceType: INSERT_STREAMING
refers to a SQL Insert operation in StarRocks. While the name contains "streaming," it is the internal designation for 
INSERT INTO
statements (such as 
INSERT INTO ... VALUES
or 
INSERT INTO ... SELECT
). It is called "streaming" because StarRocks executes these SQL commands using its internal streaming execution framework to pipe data from the source (or values list) directly to the destination tablets. Here is the distinction: * 
INSERT_STREAMING
: Corresponds to SQL Insert statements. In the 
information_schema.loads
table or 
SHOW LOAD
output, this typically appears under the 
Type
as 
INSERT
. * 
STREAM_LOAD
 (or 
STREAM
): Corresponds to the Stream Load method, which is the HTTP-based interface (typically used with 
curl
or StarRocks connectors). In short, if you see 
INSERT_STREAMING
in your logs or job metadata, the load was triggered via a SQL 
INSERT
command. References • loading/Stream_Load_transaction_interface.md • loading/StreamLoad.md • sql-reference/sql-statements/loading_unloading/STREAM_LOAD.md • loading/Loading_intro.md • loading/loading_introduction/loading_concepts.md
h
@Rocky What would have caused this?
Copy code
TransactionStatus: ABORTED
LoadJobSourceType: INSERT_STREAMING
Reason: 10.10.10.10: PTabletWriterAddSegmentRequest tablet_id 1111111 not exists
r
The error 
PTabletWriterAddSegmentRequest tablet_id [ID] not exists
indicates that a Backend (BE) received a request to write data (a segment) for a specific tablet, but it could not find an active "Writer" context for that tablet. In StarRocks, when a load job (like a Stream Load, Routine Load, or 
INSERT
) starts, it first "opens" writers on the target BEs. If the BE later receives data for a tablet but the writer is gone, this error is triggered and the transaction is aborted. Common Causes 1. Load Timeout: * The most common reason is that the load job timed out. If the FE or the BE determines the job has exceeded its allocated time, it will cancel the transaction and clean up the writers. Any subsequent data packets sent by the coordinator or other nodes for that job will result in this error. * Solution: Increase the 
query_timeout
(for 
INSERT
) or 
streaming_load_rpc_max_alive_time_sec
(for Stream Load) in the BE configuration. If using Routine Load, check 
routine_load_task_timeout_second
. 2. BE Restart or Crash: * If the BE at the IP address mentioned (
10.10.10.10
) restarted during the load, all in-memory writer contexts are lost. When the loading process attempts to send the next segment to that BE, it will report that the tablet writer "not exists." * Solution: Check the BE's 
be.out
or 
<http://be.INFO|be.INFO>
logs for restart events or crashes (OOM, etc.) around the time of the error. 3. Premature Writer Closure (Resource Issues): * If the BE encounters a critical error (e.g., Disk Full, too many tablet versions, or I/O errors) while processing an earlier part of the load, it may close the writer and return an error. If the loading coordinator doesn't stop immediately and tries to send more data, you might see this "not exists" message as a secondary error. * Solution: Search the BE logs (
<http://be.INFO|be.INFO>
) for the specific 
tablet_id
mentioned in the error to see if a more descriptive error occurred just before the abort. 4. Partition/Table Changes: * If a partition was dropped, truncated, or its distribution was changed (e.g., via dynamic partitioning) while the load was in progress, the tablet associated with that writer may have been invalidated or removed. 5. Network or RPC Issues: * High network latency or packet loss can cause the 
open
request to be delayed or fail, but if the sender proceeds to send segments anyway, the BE will reject them because the writer was never successfully initialized. Recommended Troubleshooting Steps 1. Check BE Logs: Go to the BE node 
10.10.10.10
and check 
storage/logs/be.INFO
. Search for the 
tablet_id 1111111
to find the root cause of why the writer was closed or failed to open. 2. Monitor BE Status: Run 
SHOW BACKENDS;
to ensure all BEs are 
Alive
and the 
LastStartTime
hasn't changed recently (indicating a restart). 3. Check Disk Space: Ensure the data disks on 
10.10.10.10
are not at 100% capacity. 4. Check Version Count: If you have very frequent small loads, you might hit the 
tablet_max_versions
limit (default 1000), which can cause writers to fail. Use 
SHOW TABLET 1111111;
to check the version count. References • faq/loading/Loading_faq.md • sql-reference/sql-statements/table_bucket_part_index/SHOW_TABLET.md • administration/management/BE_configuration.md • faq/Sql_faq.md • faq/operation_maintenance_faq.md • table_design/table_types/primary_key_table.md
h
@Rocky Can this be caused by table locking?
Copy code
TransactionStatus: ABORTED
LoadJobSourceType: INSERT_STREAMING
Reason: 10.10.10.10: PTabletWriterAddSegmentRequest tablet_id 1111111 not exists
r
The error 
PTabletWriterAddSegmentRequest tablet_id 1111111 not exists
typically indicates that the Backend (BE) receiving the data does not have an active "writer" context for that specific tablet and transaction. While table locking is usually not the direct cause of this specific "not exists" error, it can be an indirect factor. Here are the most likely causes: 1. Load Timeout (Most Common) If the data loading process takes too long, the BE may clean up the tablet writer to release resources. If the coordinator (FE or another BE) then tries to send more data (a "segment"), the target BE no longer recognizes the request. * Indirect Locking Link: If there is heavy lock contention (e.g., many concurrent loads or heavy compaction), the load may progress so slowly that it exceeds the timeout. * Solution: Increase the 
query_timeout
session variable for your load or adjust the BE configuration 
streaming_load_rpc_max_alive_time_sec
(default is 600 seconds). 2. BE Process Restart If the BE hosting that tablet replica crashed or was restarted during the load, all in-memory tablet writers are lost. When the load tries to resume or send more data to that BE, it will return "not exists." * Check: Check the BE 
be.out
or 
<http://be.INFO|be.INFO>
logs for any restart events around the time of the error. 3. Schema Change or Partition Operations If someone performed an 
ALTER TABLE
(schema change), 
TRUNCATE
, or 
DROP PARTITION
on the target table while the load was running, the tablets might have been replaced or deleted. * Solution: Ensure no DDL operations are running concurrently with large data loads. 4. Tablet Migration or Repair If the FE detects a disk failure or a BE becomes unresponsive, it may trigger a tablet clone or migration. If a tablet is moved while a load is trying to write to its old location, this error can occur. ────────── Is it caused by Table Locking? * Directly? No. If a table is locked (e.g., by a metadata lock in FE), you would usually see an error like 
get lock timeout
or the load would simply hang in a 
PENDING
state. * Indirectly? Yes. High lock contention can slow down the write process significantly. If the write becomes slow enough to trigger the 
streaming_load_rpc_max_alive_time_sec
timeout, the BE will close the writer, leading to the "not exists" error. Recommended Troubleshooting Steps: 1. Check BE Logs: Search the 
<http://be.INFO|be.INFO>
log on the node 
10.10.10.10
for the 
tablet_id 1111111
to see if there was a timeout or a memory/disk error. 2. Increase Timeouts: If your load is large, try:
Copy code
sql
    SET query_timeout = 3600; -- Set to 1 hour
3. Check Table Health: Run 
SHOW TABLET 1111111;
to see the current status and location of the tablet. 4. Reduce Concurrency: If you suspect lock contention, try reducing the number of concurrent load jobs or increasing 
tablet_map_shard_size
in 
be.conf
to reduce lock granularity. References • faq/loading/Loading_faq.md • faq/loading/Broker_load_faq.md • administration/management/BE_configuration.md • faq/loading/Routine_load_faq.md
h
@Rocky Can query_timeout be set in the FE or BE configuration files? Can it be set as a global variables?
r
The 
query_timeout
parameter in StarRocks is managed as a system variable, not through the 
fe.conf
or 
be.conf
configuration files. Here is the breakdown of how it can be set: 1. Can it be set as a global variable? Yes. You can set 
query_timeout
globally so that it applies to all future sessions and connections.
Copy code
sql
SET GLOBAL query_timeout = 300; -- Set global timeout to 300 seconds
Note: When set globally, the change only applies to new sessions. Existing sessions will continue to use their original timeout value. 2. Can it be set in FE or BE configuration files? No. 
query_timeout
is not a parameter in 
fe.conf
or 
be.conf
. While some internal task-related timeouts (like 
task_runs_timeout_second
) exist in 
fe.conf
, the standard timeout for user-initiated queries is handled exclusively via system variables. 3. Other ways to set 
query_timeout
 * Session Level: To change it only for your current connection:
Copy code
sql
    SET query_timeout = 600;
* Query Hint: To change it for a single specific query:
Copy code
sql
    SELECT /_+ SET_VAR(query_timeout = 60) _/ * FROM my_table;
* User Property (v3.3.3+): You can bind a specific timeout to a user:
Copy code
sql
    ALTER USER 'jack' SET PROPERTIES ('session.query_timeout' = '600');
Key Considerations: * Default Value: The default is 300 seconds (5 minutes). * Unit: The value is always in seconds. * Insert Operations: Starting from v3.4.0, 
query_timeout
no longer applies to 
INSERT
, 
UPDATE
, or 
DELETE
operations. Instead, you should use the 
insert_timeout
 variable for those tasks. References • administration/management/FE_configuration.md • administration/management/BE_configuration.md
h
@Rocky Is there a UPDATE_STREAMING for LoadJobSourceType?
r
In StarRocks, the term 
UPDATE_STREAMING
is not explicitly documented as a standard or public-facing value for 
LoadJobSourceType
in the user-level documentation (such as 
SHOW LOAD
or 
information_schema.loads
). Based on the documentation and system behavior for version 3.1 and later: 1. Publicly Documented Load Types: When querying load job information (e.g., via the 
information_schema.loads
view or 
SHOW LOAD
), the common values for the 
TYPE
or source type field are: * 
BROKER
: For Broker Load jobs. * 
INSERT
: For 
INSERT INTO
jobs. * 
STREAM
: For Stream Load jobs (visible in 
information_schema.stream_loads
). * 
ROUTINE
: For Routine Load jobs. * 
SPARK
: For Spark Load jobs. 2. UPDATE Statement Implementation: Starting from v3.1, StarRocks supports Partial Updates on Primary Key tables using the 
UPDATE
statement. These operations can be executed in two modes: * Row mode: Suitable for small batches of real-time updates. * Column mode: Suitable for updating a small number of columns across a large number of rows. Internally, these 
UPDATE
statements are often handled by the query engine and may trigger an internal ingestion-based process, but they are typically identified as 
UPDATE
or 
INSERT
in audit logs and system views rather than 
UPDATE_STREAMING
. 3. Internal Enum: In the StarRocks source code (FE), the internal 
LoadJobSourceType
enumeration includes values like 
INSERT_JOB
and 
UPDATE_JOB
. While some internal developer-facing or debug logs might refer to streaming update mechanisms (especially for Primary Key tables), 
UPDATE_STREAMING
is not a standard configuration or status value you would use in SQL commands. Summary There is no publicly documented 
UPDATE_STREAMING
value for 
LoadJobSourceType
. If you are looking to perform streaming-like updates, you should use: * Stream Load with 
partial_update: true
. * Flink/Kafka Connectors with 
sink.properties.partial_update
set to 
true
. * The 
UPDATE
 statement (which StarRocks v3.1+ optimizes for Primary Key tables). If you are seeing this term in an error message or a specific internal table, it likely refers to the internal execution mode of a Partial Update being processed as a stream of data changes for a Primary Key table. References • sql-reference/sql-statements/loading_unloading/BROKER_LOAD.md • loading/Flink-connector-starrocks.md • sql-reference/sql-statements/table_bucket_part_index/UPDATE.md • administration/management/FE_configuration.md • administration/management/BE_configuration.md
h
@Rocky What could have caused the issue?
Copy code
be/src/storage/segment_replicate_executor.cpp:131 _wait_response(replicate_tablet_infos, failed_tablet_infos) by [ReplicateToken tablet_id: 1111111, txn_id: 11111111] fail_num 2 max_fail_num 1
W20260331 11:53:33.010759 129390532884032 async_delta_writer.cpp:67] Fail to write or commit. txn_id: 11111111 tablet_id: 1111111: Internal error: PTabletWriterAddSegmentRequest tablet_id 1111111 not exists
r
The error messages you provided indicate a failure during Single Replica Load (also known as Segment Replication). What happened: 1. Mechanism: When Single Replica Load is enabled, StarRocks optimizes data ingestion by having one BE (the "primary" for that load) write and index the data into segments. It then replicates these finalized segment files to the other replicas (BEs) via RPC. 2. The Error: The coordinator BE tried to send a segment to a target BE via 
PTabletWriterAddSegmentRequest
. However, the target BE responded that 
tablet_id 1111111
does not exist in its local memory/storage. 3. Failure Threshold: 
fail_num 2 max_fail_num 1
means the replication failed on 2 replicas. Since this exceeded the allowed failure threshold (likely based on your replication factor and write quorum), the entire transaction/load task failed. Potential Causes: * Tablet Migration: The FE might believe the tablet is on a specific BE, but that BE has recently migrated the tablet elsewhere or deleted it (e.g., due to cluster balancing or the replica being marked as "bad"). * Metadata Inconsistency: There is a temporary out-of-sync state between the FE's catalog and the BE's actual local tablets. * Concurrent Schema Change/Drop: If the table or partition was being dropped, truncated, or altered (schema change) at the exact moment the load was attempting to replicate segments. * Recent BE Restart: If the target BE was recently restarted, it might still be in the process of loading tablet metadata into memory. Recommended Troubleshooting Steps: 1. Check Tablet Locations: Run the following command to see where the FE thinks the tablet should be:
Copy code
sql
    SHOW TABLET 1111111;
Compare the 
BackendId
in the output with the BEs that reported the "not exists" error in your logs. 2. Verify Tablet Health: Check if the tablet is in a healthy state:
Copy code
sql
    ADMIN SHOW REPLICA STATUS FROM table_name WHERE TabletId = 1111111;
If replicas are missing or in an error state, you may need to trigger a repair:
Copy code
sql
    ADMIN REPAIR TABLE table_name;
3. Identify Inconsistency: Check the 
be_tablets
table in 
information_schema
to see if the BE actually reports having that tablet:
Copy code
sql
    SELECT * FROM information_schema.be_tablets WHERE TABLET_ID = 1111111;
4. Workaround (If persistent): If this issue happens frequently due to a known environment instability, you can temporarily disable Single Replica Load to fall back to the standard loading process (where every BE indexes data independently):
Copy code
sql
    SET GLOBAL enable_single_replica_load = false;
(Note: This might increase CPU/IO usage on BEs during loading). If the problem persists after a few minutes (giving FE/BE time to sync), it suggests a metadata stale issue that might require a restart of the FE leader or the specific BE. References • faq/loading/Loading_faq.md • sql-reference/sql-statements/table_bucket_part_index/SHOW_TABLET.md • faq/Sql_faq.md • faq/operation_maintenance_faq.md • sql-reference/information_schema/be_tablets.md • faq/loading/Broker_load_faq.md • knowledge/trouble_shooting/fe_deadlock_problems.md
h
@Rocky Got this error. The tablet id does exist after checking with show tablet 1111111. IsSync is true. IsBad is false. IsSetBadForce is false.
Copy code
be/src/storage/segment_replicate_executor.cpp:131 _wait_response(replicate_tablet_infos, failed_tablet_infos) by [ReplicateToken tablet_id: 1111111, txn_id: 11111111] fail_num 2 max_fail_num 1
W20260331 11:53:33.010759 129390532884032 async_delta_writer.cpp:67] Fail to write or commit. txn_id: 11111111 tablet_id: 1111111: Internal error: PTabletWriterAddSegmentRequest tablet_id 1111111 not exists
r
Hello! I'm Rocky. It looks like you're encountering an error related to Single Replica Load (also known as segment replication). The error 
PTabletWriterAddSegmentRequest tablet_id 1111111 not exists
occurring within 
segment_replicate_executor.cpp
suggests that while the loading process is trying to replicate data segments from one Backend (BE) to others, the target BE reports that it doesn't recognize the tablet ID for that specific transaction. Here is a breakdown of what is likely happening and how to troubleshoot: 1. Understanding the Error When 
enable_single_replica_load
is enabled, StarRocks writes data to one replica and then the BE handles replicating those segments to the other replicas. * 
fail_num 2 max_fail_num 1
: This means the replication failed on 2 replicas, exceeding the allowed failure threshold (1), causing the entire load job to fail. * 
tablet_id 1111111 not exists
: The target BE received a request to add a segment to this tablet, but it couldn't find an active 
DeltaWriter
or the tablet itself in its local metadata for this load transaction. 2. Why 
show tablet
says it exists but the load fails Even if the tablet exists globally, the load might fail if: * Metadata Inconsistency: The FE thinks the tablet replica is on BE-A, but BE-A doesn't actually have it (or it's in a 
DELETED
state but not yet cleaned up). * Replica Migration: If a tablet migration or balance task occurred exactly when the load started, the "location" of the tablet might have changed, but the load task was still trying to talk to the old BE. * Transaction Timeout: If the 
DeltaWriter
on the target BE was closed due to a timeout or an earlier internal error, subsequent segment replication requests will fail with "not exists". 3. Troubleshooting Steps Step A: Verify Replica Health on all BEs Don't just check if the tablet exists; check if all replicas are healthy and in sync.
Copy code
sql
-- Replace 1111111 with your actual tablet_id
SHOW TABLET 1111111;
-- Take the 'DetailCmd' from the result and run it, for example:
SHOW PROC '/dbs/12345/67890/partitions/11111/111111/1111111';
In the 
SHOW PROC
output, check: * Are there replicas on BEs that were not mentioned in your error logs? * Is any replica's 
State
not 
NORMAL
or 
IsBad
true? * Are the 
Version
numbers consistent across all replicas? Step B: Check BE Logs of the Target Nodes Search the logs of the BE nodes that reported the error (not just the coordinator BE) around 
11:53:33
. Look for: * 
tablet_id: 1111111
* Any messages about 
tablet dropped
, 
clear transaction
, or 
timeout
. * Any RPC errors or "failed to open DeltaWriter". Step C: Check for Concurrent Schema Changes or Migrations Run these commands to see if any background tasks interfered:
Copy code
sql
SHOW ALTER TABLE COLUMN; -- Check for ongoing schema changes
SHOW BACKUP;             -- Check if a backup/restore is locking the tablet
Step D: Workaround If this is a persistent issue and you need the load to succeed immediately, you can temporarily disable Single Replica Load to use the standard load process (where the coordinator writes to all replicas directly):
Copy code
sql
SET GLOBAL enable_single_replica_load = false;
If the 
SHOW PROC
command reveals that one replica is consistently lagging or in a weird state, you can manually mark it as bad to force StarRocks to repair it:
Copy code
sql
ADMIN SET REPLICA STATUS PROPERTIES("tablet_id" = "1111111", "backend_id" = "FAILED_BE_ID", "status" = "bad");
Does this happen for all tables or just this specific one? Knowing if it's a one-time event or a recurring issue would help narrow it down! References • faq/loading/Loading_faq.md • faq/loading/Routine_load_faq.md
h
@Rocky Got theses in the logs. The tablet id does exist after checking with show tablet 1111111. State is NORMAL. IsSync is true. IsBad is false. IsSetBadForce is false. IsErrorState is false. Version numbers matched.
Copy code
Failed to send rpc to SyncChannnel
err=Internal error: PTabletWriterAddSegmentRequest tablet_id 1111111 not exists
Failed to sync segment SyncChannnel
err Internal error: PTabletWriterAddSegmentRequest tablet_id 1111111 not exists
be/src/storage/segment_replicate_executor.cpp:131 _wait_response(replicate_tablet_infos, failed_tablet_infos) by [ReplicateToken tablet_id: 1111111, txn_id: 11111111] fail_num 2 max_fail_num 1
W20260331 11:53:33.010759 129390532884032 async_delta_writer.cpp:67] Fail to write or commit. txn_id: 11111111 tablet_id: 1111111: Internal error: PTabletWriterAddSegmentRequest tablet_id 1111111 not exists
get clear transaction task task, signature:256011111, txn_id: 11111111, partition id size: 0
Clearing transaction task txn_id: 11111111
Cleared transaction task txn_id: 11111111
finish to clear transaction task. signature:256011111, txn_id: 11111111
r
This error message indicates a failure within the Segment Replication (also known as Single Replica Loading) process in StarRocks. The error 
PTabletWriterAddSegmentRequest tablet_id 1111111 not exists
means that while the Primary replica of the tablet successfully wrote a data segment and tried to replicate it to the Secondary replicas, the target Backend (BE) reported that it does not have that tablet ID in its memory or disk. Why this happens When Segment Replication is enabled (default for Primary Key tables or when 
replicated_storage
is set to 
true
), only one replica (the primary) processes the data and then sends the finished segment files to others via the 
SyncChannel
. If a secondary BE receives this RPC but cannot find the tablet, the request fails. This usually happens due to: 1. Metadata Inconsistency: The FE thinks a BE should have a replica of tablet 
1111111
, but that BE has either dropped it, failed to create it, or it was recently migrated/moved. 2. Stale Tablet Info: A tablet migration or balancing task might have recently moved the tablet, and the Coordinator BE is still trying to send data to the old location. 3. Failed Tablet Initialization: The tablet might have failed to initialize on that specific BE during a schema change or partition creation. Troubleshooting Steps 1. Check all replicas of the tablet While 
SHOW TABLET 1111111
shows a general "NORMAL" state, you need to look at the status of every individual replica across all BEs. 1. Run 
SHOW TABLET 1111111;
. 2. Copy the command in the DetailCmd column (it looks like 
SHOW PROC '/dbs/1001/tablets/1111111';
). 3. Execute that command. 4. Check the 
State
, 
IsBad
, and 
Version
for every 
BackendId
listed. Look for any BE where the tablet might be missing or in an 
ERROR
state. 2. Identify the "Missing" BE Check your BE logs for the IP/ID of the node that returned the 
not exists
error. Cross-reference this with the list from Step 1. If a BE is listed in the FE metadata but doesn't actually host the tablet, that is the source of the inconsistency. 3. Force a Repair If you find a replica that is lagging or inconsistent on a specific BE, you can force StarRocks to recreate it:
Copy code
sql
-- Replace with your actual tablet_id and the backend_id that is failing
ADMIN SET REPLICA STATUS PROPERTIES("tablet_id" = "1111111", "backend_id" = "10002", "status" = "bad");
After marking it as 
bad
, the FE will automatically schedule a clone task to restore a healthy replica from the remaining good ones. 4. Temporary Workaround (If persistent) If this is a recurring issue on a specific table and impacting production, you can disable Segment Replication for that table to revert to the traditional "replicate-to-all" loading mode:
Copy code
sql
ALTER TABLE <table_name> SET ("replicated_storage" = "false");
Note: This might increase CPU usage during loads as every replica will now process the data independently. Recommendation: If the 
DetailCmd
shows all replicas are 
NORMAL
and versions match, but the error persists, it is likely a transient race condition during a tablet migration. Restarting the BE that reports the "not exists" error can often clear the internal metadata cache. References • faq/loading/Loading_faq.md • administration/management/resource_management/Replica.md • faq/operation_maintenance_faq.md • sql-reference/sql-statements/table_bucket_part_index/SHOW_TABLET.md
h
@Rocky This error happened where the CPU and Memory usage was high and there were many select/insert/update running in parallel. What can be done to reduce this error?
Copy code
Failed to send rpc to SyncChannnel
err=Internal error: PTabletWriterAddSegmentRequest tablet_id 1111111 not exists
Failed to sync segment SyncChannnel
err Internal error: PTabletWriterAddSegmentRequest tablet_id 1111111 not exists
be/src/storage/segment_replicate_executor.cpp:131 _wait_response(replicate_tablet_infos, failed_tablet_infos) by [ReplicateToken tablet_id: 1111111, txn_id: 11111111] fail_num 2 max_fail_num 1
W20260331 11:53:33.010759 129390532884032 async_delta_writer.cpp:67] Fail to write or commit. txn_id: 11111111 tablet_id: 1111111: Internal error: PTabletWriterAddSegmentRequest tablet_id 1111111 not exists
get clear transaction task task, signature:256011111, txn_id: 11111111, partition id size: 0
Clearing transaction task txn_id: 11111111
Cleared transaction task txn_id: 11111111
finish to clear transaction task. signature:256011111, txn_id: 11111111
r
This error is related to the Single Leader Replication (also known as 
replicated_storage
) feature in StarRocks. When 
replicated_storage
is set to 
true
(which is the default in v3.0+), data is sent to a primary replica BE, which then replicates the data segments to other BEs (secondary replicas) via a component called 
SyncChannel
. The error 
PTabletWriterAddSegmentRequest tablet_id 1111111 not exists
indicates that the primary BE tried to send a data segment to a secondary BE, but the secondary BE's "Tablet Writer" for that specific tablet was either never successfully opened or was closed/cleaned up (often due to a timeout or failure during the initialization phase). Under high CPU and Memory usage with many parallel operations, the internal RPCs used to coordinate these writers are delayed or time out, leading to this inconsistency. Recommended Solutions 1. Increase RPC Timeouts The most direct way to mitigate this under high load is to give the BEs more time to initialize tablet writers. * Parameter: 
tablet_writer_open_rpc_timeout_sec
* Location: 
be.conf
* Action: Increase the value (e.g., from the default 
60
or 
300
to 
600
). * Restart: Required for each BE. 2. Reduce Lock Contention If you have many tablets and high concurrency, lock contention on the tablet map can cause delays. * Parameter: 
tablet_map_shard_size
* Location: 
be.conf
* Action: Increase this value to 
512
(default is usually 
4
or 
8
). This helps parallelize access to tablet metadata. * Restart: Required for each BE. 3. Optimize Loading Strategy Many small, parallel 
INSERT
or 
UPDATE
operations consume significant CPU and create many small versions, leading to heavy metadata pressure and compaction overhead. * Batching: Combine small load jobs into larger batches to reduce the frequency of transactions. * Concurrency: Limit the number of parallel load/update tasks to keep CPU and Memory usage within a stable range (e.g., below 80%). 4. Fallback: Disable Single Leader Replication If the resource pressure cannot be reduced and the errors persist, you can revert specific tables to the older replication method (Leaderless Replication), where the loading client writes to all replicas simultaneously. This removes the "SyncChannel" bottleneck but may increase the CPU cost of the load itself. * Command:
Copy code
sql
    ALTER TABLE <table_name> SET ("replicated_storage" = "false");
5. Check for Tablet Health Sometimes this error occurs if replicas are already in an inconsistent state. Check the health of the tablets mentioned in the logs:
Copy code
sql
ADMIN SHOW REPLICA STATUS FROM <table_name> WHERE TabletId = 1111111;
If any replicas are marked as 
IsBad = true
, they may need to be repaired or dropped for StarRocks to clone fresh copies. References • sql-reference/sql-statements/table_bucket_part_index/CREATE_TABLE.md • knowledge/usage/best-practice.md • administration/management/resource_management/Replica.md • administration/management/FE_configuration.md • faq/operation_maintenance_faq.md
h
@Rocky Is it ok to configure like below for a cluster of 3 FE (each FE has 8 vCPU, 16GB memory) and dedicated 3 BE nodes (each BE has 16 vCPU, 64GB memory) that has heavy workload (using insert/update/delete/select) in parallel (high CPU and memory usage)?
Copy code
tablet_writer_open_rpc_timeout_sec = 600
tablet_map_shard_size = 512
r
Based on the hardware specs (3 FE with 16GB RAM, 3 BE with 64GB RAM) and the heavy parallel workload you've described, here is a breakdown of the configurations you've proposed: 1. 
tablet_writer_open_rpc_timeout_sec = 600
 * What it does: This sets the timeout for the RPC that opens a tablet writer on a BE node during a load process. * Is it okay? Yes, it is safe to increase this value to 600 seconds (10 minutes) for heavy workloads. * Context: The default value is typically 300 seconds (since v3.2.0). Under heavy CPU and memory pressure, BE nodes may respond more slowly to writer-open requests because they are busy flushing data or managing locks. Increasing this to 600s acts as a "buffer" to prevent load jobs from failing prematurely due to transient pressure. * Note: If your load jobs frequently hit this 10-minute timeout, it usually indicates a deeper bottleneck in I/O or CPU, and increasing the timeout further is just a "band-aid." 2. 
tablet_map_shard_size = 512
 * What it does: This partitions the internal "tablet map" lock into shards to reduce lock contention when many threads are trying to access or update tablet metadata simultaneously. * Is it okay? Yes, but it depends on your StarRocks version. * In older versions: The default was often very low (sometimes 1 or 4). In those cases, setting it to 512 is a significant improvement for concurrency. * In newer versions (v3.2.0+): The default has been increased to 1024. If you are on a recent version, setting it to 512 would actually be a reduction in sharding. * Recommendation: For your workload, ensure this value is a power of two. If you are on v3.2 or later, I recommend leaving it at the default 1024 or even increasing it to 2048 if you have tens of thousands of tablets per BE. Additional Recommendations for Your Hardware Your 64GB BE nodes are relatively light for "heavy workloads" involving updates and deletes (which are resource-intensive in StarRocks). You should also consider these adjustments: 1. Memory Management: With 64GB RAM, ensure you monitor the BE memory usage. If you see "Memory Limit Exceeded" errors, you may need to tune 
mem_limit
(default is 90%) or increase the hardware. 2. Concurrency: Since you have parallel Insert/Update/Delete operations, keep an eye on Compaction. Heavy writes trigger heavy compaction. Check 
compaction_max_running_iteration
and 
max_compaction_threads
in BE config to ensure compaction doesn't starve your query resources. 3. Transaction Management: Since you have high concurrency, you might also want to check 
txn_map_shard_size
(default 128) and 
txn_shard_size
(default 1024) to ensure transaction locks aren't becoming a bottleneck. Summary: Your proposed changes are safe and generally helpful for high-pressure environments, but check your StarRocks version to ensure 
tablet_map_shard_size
is an increase rather than a decrease from the default. References • administration/management/BE_configuration.md • faq/Sql_faq.md • administration/management/FE_configuration.md
h
@Rocky What is this folder for? Is there a process to clean up the files in it?
Copy code
/opt/starrocks/fe/temp_dir/backup
r
The folder 
/opt/starrocks/fe/temp_dir/backup
is used as a temporary staging area by the Frontend (FE) during 
BACKUP
 and 
RESTORE
 operations. What is this folder for? When you run a backup or restore task, StarRocks FE often needs to store intermediate metadata or temporary files locally before moving them to the remote repository (like S3, HDFS, or MinIO) or after downloading them from one. * The parent directory is controlled by the FE configuration parameter 
tmp_dir
(which defaults to 
FE_HOME/temp_dir
). * The 
backup
subfolder specifically holds files related to these snapshot/recovery procedures. Is there a process to clean up the files in it? 1. Automatic Cleanup: Under normal circumstances, StarRocks is designed to delete these temporary files automatically once the backup or restore procedure finishes (whether it succeeds or fails). 2. Manual Cleanup: If you see files lingering in this folder, it usually means a backup/restore process was interrupted, crashed, or encountered an error that prevented the cleanup hook from running. How to safely clean it up: * Step 1: Check for active tasks. Before deleting anything, ensure there are no backup or restore jobs currently running. Run these commands in your MySQL client:
Copy code
sql
    SHOW BACKUP;
    SHOW RESTORE;
Look at the 
State
column. If any task is 
PENDING
, 
SNAPSHOTING
, or 
UPLOADING
, do not delete the files. * Step 2: Manual deletion. If there are no active tasks and the files are old, you can safely remove them manually using the standard Linux command:
Copy code
bash
    rm -rf /opt/starrocks/fe/temp_dir/backup/*
Note: Do not delete the 
backup
folder itself, just the contents inside it. Summary of FE Configuration If you want to move this temporary directory to a disk with more space (to avoid "disk full" issues during large backups), you can modify your `fe.conf`:
Copy code
properties
# Example: changing the temp directory path
tmp_dir = /path/to/larger/disk/temp_dir
A restart of the FE is required for this change to take effect. References • administration/management/FE_configuration.md • faq/operation_maintenance_faq.md • faq/Others.md • knowledge/admin_and_operation/shared_data_garbage_cleanup.md • administration/Meta_recovery.md • administration/management/BE_configuration.md
h
@Rocky We got this error when submitting stream load
Copy code
Error while loading data: HTTPConnectionPool(host='10.0.10.10', port=8040): Max retries exceeded with url: /api/db/table/_stream_load (Caused by ConnectTimeoutError(<urllib3.connection.HTTPConnection object at 0x7c14cbc11111>, 'Connection to 10.10.10.10 timed out. (connect timeout=60)'))
r
This error indicates a network connectivity issue between your client (where you are running the load script) and the StarRocks Backend (BE) node at 
10.0.10.10
on port 
8040
. Why this happens Stream Load in StarRocks follows a two-step redirection process: 1. Request to FE: Your client sends the initial request to the Frontend (FE) (typically port 8030). 2. Redirection to BE: The FE selects an available BE node and sends an HTTP 307 redirect back to your client. 3. Connection to BE: Your client automatically tries to connect to that BE (in your case 
10.0.10.10:8040
) to stream the actual data. The error 
ConnectTimeoutError
means your client successfully talked to the FE, but it cannot reach the BE it was redirected to. Troubleshooting Steps 1. Verify Connectivity From the machine where you are submitting the load, try to reach the BE port manually:
Copy code
bash
telnet 10.0.10.10 8040
# OR
curl -I <http://10.0.10.10:8040/api/health>
* If this fails, there is a firewall, security group (AWS/GCP), or routing issue blocking traffic to port 8040 on your BE nodes. 2. Check BE Status Ensure the BE is actually up and running. Run this in your MySQL client:
Copy code
sql
SHOW BACKENDS\G
* Check the 
Alive
column. If it is 
false
, the BE is down and cannot accept loads. * Check the 
IP
column. Ensure the IP listed there is reachable from your client. 3. Internal vs. External IP (Common in Cloud/K8s) If your StarRocks cluster is in a private network (like a VPC or Kubernetes) and your client is outside: * The FE will redirect you to the internal IP of the BE (e.g., 
10.0.10.10
). * If your client cannot route to that internal IP, the connection will time out. * Solution: You must ensure your client has a network path to the BEs, or run the load from a machine inside the same network. 4. Configure 
priority_networks
 If your BE nodes have multiple network interfaces (e.g., a public and a private one), they might be registering the "wrong" IP with the FE. * Check 
be.conf
for the 
priority_networks
parameter. * It should be set to the CIDR of the network you want StarRocks to use (e.g., 
priority_networks = 10.0.10.0/24
). 5. Port Conflicts Confirm that 
be_http_port
is actually 
8040
in your 
be.conf
. If you changed it to something else, ensure that port is open. Summary The FE told your client: "Please send the data to 10.0.10.10:8040", but your client couldn't find a way to get there. Focus on opening port 8040 in your network security settings or ensuring the client can route to the BE's IP address. References • data_source/data_cache.md • administration/management/BE_configuration.md • using_starrocks/caching/block_cache.md
h
@Rocky We got this error after submitting a stream load directly to the BE IP provided by the FE. There are no network issues
Copy code
Error while loading data: HTTPConnectionPool(host='10.0.10.10', port=8040): Max retries exceeded with url: /api/db/table/_stream_load (Caused by ConnectTimeoutError(<urllib3.connection.HTTPConnection object at 0x7c14cbc11111>, 'Connection to 10.10.10.10 timed out. (connect timeout=60)'))
r
The error message you provided contains a very specific clue: a mismatch between the target IP you requested (
10.0.10.10
) and the actual IP the connection timed out on (
10.10.10.10
).
Copy code
text
HTTPConnectionPool(host='10.0.10.10', port=8040): ... 
Caused by ConnectTimeoutError(..., 'Connection to 10.10.10.10 timed out.')
Even if you believe there are no network issues, a 
ConnectTimeoutError
at the TCP level means your client cannot reach the destination IP/Port. Here is why this is happening and how to fix it: 1. The "Redirection" Trap When you submit a Stream Load, the following sequence usually occurs: 1. Your client sends a request to a StarRocks node (could be an FE or a BE). 2. The node determines which BE should handle the load and sends an HTTP 307 Redirect back to your client with the specific BE's IP and port (
8040
). 3. Your client library (like Python 
requests
) automatically follows that redirect. The Issue: The Frontend (FE) is likely telling your client to go to 
10.10.10.10
, which is likely an internal/private IP that the BE uses to communicate within the cluster. If your client is running on a machine outside that specific private network (e.g., a different VPC, a local laptop, or a different subnet), it will time out trying to reach that internal IP. 2. Why the IP is "Wrong" StarRocks BEs register themselves with the FE using the IP they find on their network interface. If a BE has multiple interfaces or is running in a container/K8s: * It might pick a "management" or "internal" IP that isn't routable from your client. * The FE then passes this "internal" IP to your client during the redirect. 3. How to Fix This Option A: Use 
priority_networks
(Recommended) You need to tell the StarRocks BEs which network they should "advertise" to the FE. 1. Edit 
be.conf
on all BE nodes. 2. Set the 
priority_networks
parameter to the CIDR of the network that is reachable by your client.
Copy code
properties
    # Example: If your reachable network is 10.0.10.x
    priority_networks = 10.0.10.0/24
3. Restart the BEs. 4. Verify the IP in the cluster by running 
SHOW BACKENDS;
in your MySQL client. The 
IP
column should now show the reachable 
10.0.10.x
addresses. Option B: Check Firewall/Security Groups If the IPs are actually correct and routable, the timeout on port 8040 specifically means: * The port 
8040
is blocked by a firewall (iptables, firewalld) on the BE node. * The Cloud Security Group (AWS/GCP/Azure) does not allow inbound traffic on 
8040
. * Test this: From your client machine, run 
telnet 10.10.10.10 8040
. If it hangs, the network path is blocked. Option C: Client-side Routing If you cannot change the StarRocks configuration, ensure the machine running the load has a network route to the 
10.10.10.x
subnet (e.g., via a VPN or VPC Peering). Summary Checklist * [ ] Does 
SHOW BACKENDS;
show the 
10.10.10.10
IP? (If yes, FE is redirecting you there). * [ ] Can your client machine 
ping 10.10.10.10
? * [ ] Can your client machine 
telnet 10.10.10.10 8040
? Rocky's Pro Tip: If you are using 
curl
, use the 
-L
or 
--location-trusted
flag. If you are using a Python script, ensure you aren't behind a proxy that is intercepting the redirect incorrectly. References • quick_start/shared-data.md • quick_start/helm.md • loading/StreamLoad.md • administration/management/FE_configuration.md • administration/management/BE_configuration.md • faq/loading/Stream_load_faq.md
h
@Rocky We got this error after submitting a stream load directly to the BE IP provided by the FE. There are no network issues
Copy code
Error while loading data: HTTPConnectionPool(host='10.10.10.10', port=8040): Max retries exceeded with url: /api/db/table/_stream_load (Caused by ConnectTimeoutError(<urllib3.connection.HTTPConnection object at 0x7c14cbc11111>, 'Connection to 10.10.10.10 timed out. (connect timeout=60)'))
r
If you are seeing a 
ConnectTimeoutError
for 
10.10.10.10:8040
, it means your client (the script/machine submitting the load) failed to establish a TCP handshake with the StarRocks Backend (BE) within the 60-second limit. Even if you believe there are no network issues, a "Timeout" (as opposed to "Connection Refused") means the client sent a request and received nothing back. This is almost always caused by one of the following environmental or configuration factors: 1. The "Listen" vs. "Routable" IP StarRocks BEs bind to specific network interfaces. If a BE has multiple IPs (e.g., an internal 10.x.x.x and a management IP), it might only be listening on the one identified by 
priority_networks
. * The Issue: If the BE is listening on 
10.10.10.10
but you are trying to reach it from a network that doesn't have a direct route to that specific subnet, it will time out. * Verification: On the BE node itself, run:
Copy code
bash
    netstat -antp | grep 8040
Confirm it shows 
LISTEN
and check which IP it is bound to (e.g., 
0.0.0.0:8040
or a specific IP). 2. Environment Proxies (Common Culprit) If your client machine has 
http_proxy
or 
https_proxy
environment variables set, 
urllib3
(used by Python's 
requests
) will attempt to route the connection through that proxy. * The Issue: If your proxy cannot reach the internal StarRocks IP 
10.10.10.10
, the proxy will time out, which manifests as a 
ConnectTimeoutError
in your script. * Fix: Try running your load with the proxy disabled:
Copy code
bash
    unset http_proxy https_proxy
    # then run your load script
3. Port-Level Blocking (Security Groups/Firewalls) Ping uses ICMP, but Stream Load uses TCP. Passing a 
ping
test does not mean port 8040 is open. * The Issue: A firewall (iptables/firewalld) on the BE or a Cloud Security Group (AWS/GCP) might allow traffic to the FE (8030) but block traffic to the BE (8040). * Verification: From your client machine, run:
Copy code
bash
    telnet 10.10.10.10 8040
    # OR
    curl -v <http://10.10.10.10:8040/api/health>
If these commands hang, the port is blocked. 4. 
priority_networks
Mismatch If you are submitting "directly to the BE IP provided by the FE," ensure that the IP the FE gave you is actually the one you expect. * Run 
SHOW BACKENDS;
in your MySQL client. * If the 
IP
column shows an address that is not routable from your client's location (like a Docker internal IP or a K8s Pod IP), you must update the 
priority_networks
in 
be.conf
to a CIDR that matches your routable network and restart the BE. 5. BE is Overloaded or Stuck If the BE is experiencing a "Stop-the-world" GC pause or the CPU is at 100%, the TCP listen queue might be full, causing new connection attempts to be dropped. * Check the BE logs (
be/log/be.INFO
and 
be/log/be.out
) for any crash signatures or "High heartbeat" warnings. Recommended Next Step Run this command from your client machine: 
curl -I <http://10.10.10.10:8040/api/health>
* If it times out: Your network path to port 8040 is definitely blocked (check firewalls/proxies). * If it returns 200 OK: The network is fine, and the issue might be related to the specific client library handling of 307 redirects. In this case, ensure your script is not accidentally trying to redirect back to a non-routable FE address. References • administration/management/BE_configuration.md • faq/loading/Stream_load_faq.md • loading/StreamLoad.md • faq/Deploy_faq.md • administration/management/BE_blacklist.md References
h
@Rocky We got this error after submitting a stream load directly to the BE IP provided by the FE. This error only happens sometimes. There are no network issues
Copy code
Error while loading data: HTTPConnectionPool(host='10.10.10.10', port=8040): Max retries exceeded with url: /api/db/table/_stream_load (Caused by ConnectTimeoutError(<urllib3.connection.HTTPConnection object at 0x7c14cbc11111>, 'Connection to 10.10.10.10 timed out. (connect timeout=60)'))
r
If you are experiencing intermittent 
ConnectTimeoutError
on port 8040 (the BE HTTP port) and you have already confirmed that the IP is correct and basic network connectivity (ping) is fine, the issue is likely due to resource saturation or queueing on the Backend (BE) node or the client. A "Connect Timeout" (especially one as long as 60 seconds) indicates that your client sent a TCP 
SYN
packet, but the BE did not complete the handshake in time. Here are the most common reasons this happens intermittently: 1. BE HTTP Thread Exhaustion The StarRocks BE has a fixed number of threads dedicated to handling HTTP requests (including Stream Load). * The Parameter: 
be_http_num_workers
(Default: 48). * The Issue: If you are running high-concurrency Stream Loads or many concurrent 
INSERT INTO ... SELECT
jobs, these 48 threads can become fully occupied. When all threads are busy, new connection attempts are placed in the OS-level TCP backlog. If the backlog is full or the threads don't free up quickly, the connection will time out. * Solution: Increase the number of HTTP worker threads in your 
be.conf
and restart the BE:
Copy code
properties
    be_http_num_workers = 128
2. OS-level TCP Backlog Limits If you have bursts of many connections, the Linux kernel might drop incoming connection requests if the "Listen" queue is full. * Check: Run this command on the BE node during a period of failure:
Copy code
bash
    netstat -s | grep -i listen
    # Look for "SYNs to LISTEN sockets dropped" or "times the listen queue of a networking endpoint was full"
* Solution: Increase the system's maximum connection backlog.
Copy code
bash
    # Check current limits
    sysctl net.core.somaxconn
    sysctl net.ipv4.tcp_max_syn_backlog

    # Increase them (e.g., to 2048 or higher)
    sysctl -w net.core.somaxconn=2048
    sysctl -w net.ipv4.tcp_max_syn_backlog=2048
3. bRPC Connection Type (High Concurrency) For environments with extremely high concurrency, the default connection management can sometimes lead to bottlenecks. * Solution: Change the bRPC connection type to 
pooled
in 
be.conf
to improve RPC efficiency:
Copy code
properties
    brpc_connection_type = pooled
4. Client-side Connection Pool Maxing Out The error message mentions 
HTTPConnectionPool(host='10.10.10.10', port=8040): Max retries exceeded
. This can happen if your client-side script (e.g., a Python script using 
requests
) is reaching its own internal limit of concurrent connections or retries. * Troubleshooting: Ensure your client code is reusing connections (using a 
requests.Session()
object) rather than opening a new connection for every single chunk or request. * Check Proxies: If your environment has a transparent proxy or an 
http_proxy
environment variable set, the intermittent timeout might be happening at the proxy layer. 5. High BE CPU / IO Wait If the BE node is experiencing extreme CPU load or disk I/O wait (due to heavy compaction or complex queries), the kernel may be slow to process the TCP stack, leading to intermittent handshake timeouts. * Check: Monitor 
top
or 
iostat
on the BE during the failures. If the CPU load is consistently 100%, you may need to scale out your cluster or reduce the loading frequency. Summary Checklist 1. Increase 
be_http_num_workers
 to 128 or 256 in 
be.conf
. 2. Verify OS backlog with 
netstat -s
to see if packets are being dropped by the kernel. 3. Check for "High Heartbeat" warnings in 
<http://be.INFO|be.INFO>
logs, which indicate the BE is too busy to respond promptly. 4. If using Kubernetes, check if CPU Throttling is occurring on the BE pods. References • administration/management/BE_configuration.md • faq/operation_maintenance_faq.md
h
@Rocky There are many files related to backups in /opt/starrocks/fe/temp_dir. Why is this happening and if there is a automatic process to clean up the files?
r
The accumulation of files in the 
/opt/starrocks/fe/temp_dir
directory is typically related to the BACKUP and RESTORE operations. Here is a breakdown of why this happens and how to manage it. 1. Why is this happening? The 
temp_dir
(configured by the FE parameter 
tmp_dir
) is used as a staging area to store temporary metadata and snapshot files during the backup and restore processes. * Expected Behavior: StarRocks is designed to automatically delete these files once a backup or restore job finishes (either successfully or through a handled failure). * The Cause of Accumulation: If backup or restore jobs are interrupted, cancelled, or time out (due to network issues with remote storage or FE restarts), the cleanup logic may not trigger correctly. Over time, these orphaned temporary files accumulate and consume disk space. 2. Is there an automatic cleanup process? Unlike the 
trash
directory (which has the 
catalog_trash_expire_second
parameter for periodic cleanup), there is no background thread that periodically scans and purges the 
temp_dir
. The cleanup is strictly "procedural"—it is tied to the lifecycle of a specific job. If the job's lifecycle is broken (e.g., a hard crash of the FE), the files remain. 3. How to fix and clean up You can resolve this by following these steps: Step 1: Check for active jobs Before deleting anything, ensure there are no legitimate backup or restore tasks currently running.
Copy code
sql
SHOW BACKUP;
SHOW RESTORE;
If you see jobs in a 
PENDING
, 
SNAPSHOTING
, or 
UPLOADING
state that have been stuck for a long time, you should cancel them:
Copy code
sql
CANCEL BACKUP FROM <database_name>;
CANCEL RESTORE FROM <database_name>;
Step 2: Manual Cleanup If no jobs are active, it is perfectly safe to manually delete the contents of the 
temp_dir
.
Copy code
bash
# Navigate to the directory and remove old files
rm -rf /opt/starrocks/fe/temp_dir/*
Note: Do not delete the directory itself, only its contents. Step 3: Preventative Tuning If your backup jobs frequently time out and leave files behind, you can increase the timeout limit in `fe.conf`: * Parameter: 
backup_job_default_timeout_ms
* Default: 
86400000
(24 hours). * Action: If your datasets are massive and the remote storage is slow, increasing this can help jobs finish cleanly (and thus trigger the auto-delete). Summary Recommendation If you notice this happening frequently, it is common practice for administrators to set up a simple cron job to delete files in that directory that are older than a few days, provided no backups are scheduled during that window.
Copy code
bash
# Example: delete files older than 3 days in the temp_dir
find /opt/starrocks/fe/temp_dir/ -type f -mtime +3 -delete
References • administration/management/FE_configuration.md • administration/Meta_recovery.md • faq/operation_maintenance_faq.md
h
@Rocky There are many files related to backups in /opt/starrocks/fe/temp_dir. But the backup cron is not enabled on this FE but the other one. Why is this happening and if there is a automatic process to clean up the files?
r
The reason you see backup files in the 
temp_dir
of an FE where no "backup cron" is enabled is due to how StarRocks handles the Leader FE role and cluster-wide operations. 1. The Leader FE Performs the Work In a StarRocks cluster, the 
BACKUP
and 
RESTORE
operations are global, cluster-wide tasks. Even if your cron job connects to "the other FE" (e.g., FE-B) to issue the SQL command, the Leader FE is the one that actually coordinates the job, generates metadata snapshots, and writes them to its local 
temp_dir
before uploading them to your remote storage (HDFS/S3). * Check the Leader: Run 
SHOW FRONTENDS;
in your MySQL client. * If the FE you are currently looking at has 
IsMaster
(or 
Role: LEADER
) set to 
true
, it is the one responsible for the backup work, regardless of where the SQL command originated. 2. Leader Role Transitions If your cluster has had a Leader election recently (e.g., due to a restart or network blip), the old Leader may have been in the middle of a backup job when it lost its leadership. * When a Leader fails or steps down, the temporary files created in its 
temp_dir
are often orphaned. * The new Leader will start its own process, but it cannot "reach back" and clean up the filesystem of the previous Leader. 3. Cleanup Logic As mentioned previously, StarRocks' cleanup of 
temp_dir
is event-driven, not time-driven. * Success/Failure: The files are deleted only when the FE successfully finishes the job (or catches a specific error). * Hard Crashes/Orphaned Files: If the FE process is killed (
kill -9
) or the node loses leadership, the "cleanup" code path is never executed. Because there is no background "garbage collector" thread for this directory, the files stay there indefinitely. Recommended Actions 1. Identify the Leader: Confirm if the FE with the files is (or was) the Leader. 2. Verify Job Status: Run 
SHOW BACKUP;
and 
SHOW RESTORE;
. If there are any 
CANCELLED
or 
FINISHED
jobs that didn't clean up, or if there are very old 
SNAPSHOTING
jobs, they are the source. 3. Manual Purge: If you have confirmed no backups are currently running (
SHOW BACKUP
returns nothing in progress), it is safe to manually delete the contents:
Copy code
bash
    rm -rf /opt/starrocks/fe/temp_dir/*
4. Cron for Cleanup: Since StarRocks does not provide an automatic periodic cleanup for this specific directory, it is a best practice to add a simple maintenance cron to all FE nodes to delete files older than 7 days:
Copy code
bash
    find /opt/starrocks/fe/temp_dir/ -type f -mtime +7 -delete
Summary The files exist there because that FE is (or was) the Leader when a backup was initiated. The "cron" only tells the cluster to start; the Leader decides where the temporary metadata work happens. References • administration/management/FE_configuration.md • administration/Meta_recovery.md
h
@Rocky, I got this error on BE. Would you know what it means and how to fix it?
Copy code
http_client.cpp:173] fail to execute HTTP client, errmsg=Failed to connect to 10.10.10.10 port 8000 after 7151 ms: Couldn't connect to server
r
The error message 
http_client.cpp:173] fail to execute HTTP client, errmsg=Failed to connect to 10.10.10.10 port 8000
indicates that your StarRocks Backend (BE) is attempting to make an HTTP request to a service at 
10.10.10.10:8000
, but the connection is being refused or timing out at the TCP level. In the context of StarRocks, Port 8000 is the default port for the StarRocks Broker process. However, since the error comes from 
http_client.cpp
, it means the BE is specifically trying to communicate via HTTP, not just Thrift (the Broker's primary protocol). Here are the most likely causes and how to fix them: 1. Broker Service is Down or Unreachable If you are using a Broker to load data from HDFS or Cloud Storage (e.g., via 
LOAD LABEL
or 
INSERT INTO ... SELECT
with a Broker), the BE needs to communicate with it. * Check Status: Run the following SQL command in your StarRocks MySQL client:
Copy code
sql
    SHOW BROKER;
* Action: Check if the Broker at 
10.10.10.10
shows 
Alive: true
. If it is 
false
, the Broker process on that machine has likely crashed or wasn't started. Log into that machine and check the Broker logs (
apache_hdfs_broker.out
or 
broker.log
). 2. Misconfigured Storage Endpoint (S3 / MinIO / OSS) If you are using External Tables, Storage Volumes, or Backup/Restore to an object store (like MinIO), you may have accidentally specified port 8000 in your endpoint URL. * The Issue: MinIO's API typically runs on port 9000 (its Console often runs on 9001). If your SQL command or configuration used 
<http://10.10.10.10:8000>
, the BE will try to connect there and fail. * Action: Check your 
CREATE EXTERNAL TABLE
or 
BACKUP
statement. Ensure the 
aws.s3.endpoint
or similar property points to the correct API port. 3. Firewall or Network Security Groups Even if the service is running, the BE node might be blocked from reaching port 8000 on the target IP. * Test Connectivity: From the BE node where the error occurs, try to manually connect to that port:
Copy code
bash
    telnet 10.10.10.10 8000
    # OR
    curl -v <http://10.10.10.10:8000>
* Action: If 
telnet
fails with "Connection refused" or "Timeout," you must open port 8000 in your OS firewall (`iptables`/`firewalld`) or your cloud provider's Security Group. 4. Elasticsearch External Tables If you are querying Elasticsearch (ES) through StarRocks, the BE acts as an HTTP client to the ES nodes. * The Issue: ES typically defaults to port 9200. If your ES cluster or a proxy was manually configured to port 8000, ensure the service is actually listening. * Action: Verify the 
hosts
property in your ES external table DDL:
Copy code
sql
    CREATE EXTERNAL TABLE es_table (...)
    ENGINE=ELASTICSEARCH
    PROPERTIES (
        "elasticsearch.hosts" = "<http://10.10.10.10:8000>",
        ...
    );
5. Lakehouse REST Catalog If you are using a REST Catalog for Iceberg or another data lake format, it may be running on port 8000. * Action: Verify that the catalog service is healthy and that the 
catalog_uri
in your StarRocks Catalog configuration is correct. Summary Checklist: 1. Run 
SHOW BROKER;
to see if a Broker is configured on that port and if it is "Alive." 2. Check if the IP 
10.10.10.10
corresponds to a MinIO or Object Storage server and verify its actual API port (usually 9000). 3. Check for Firewalls between the BE and the target IP. 4. Search your BE 
<http://be.INFO|be.INFO>
logs around the time of the error for a "query_id" or "load_id" to identify which specific SQL command or background job is triggering this connection attempt. References • deployment/environment_configurations.md • sql-reference/sql-statements/cluster-management/nodes_processes/SHOW_BROKER.md
h
@Rocky We got this grafana alert. Would you know what to check?
Copy code
BE tablet clone failed on [no value]
r
The alert 
BE tablet clone failed
 indicates that the cluster is unable to successfully copy a data shard (tablet) from one Backend (BE) node to another. In StarRocks, cloning is the mechanism used for: * Replica Repair: If a BE goes down or a disk fails, the cluster "clones" the missing replicas onto healthy nodes. * Balancing: If one BE has significantly more data than others, the cluster "clones" tablets to underutilized nodes to balance the load. 1. Why did the alert trigger? The alert is usually based on the metric 
starrocks_be_engine_requests_total{type="clone", status="failed"}
. Common reasons for failure include: * Network Issues: High latency or packet loss between the source BE and destination BE. * Disk Issues: The destination BE's disk is full, in read-only mode, or experiencing I/O errors. * Resource Exhaustion: The BE is under heavy CPU/Memory load and the clone task timed out. * Version Mismatch: The source replica has a version that is too old or "cold" (already moved to remote storage) and cannot be easily cloned. 2. How to Troubleshoot Since your alert shows 
[no value]
, you first need to identify which tablets and nodes are involved. Step A: Check the Clone History The most direct way to see why clones are failing is to check the FE's internal scheduling records. Run this in your MySQL client:
Copy code
sql
SHOW PROC '/cluster_balance/history_tablets';
* What to look for: Find rows where the 
State
is 
CANCELLED
, 
TIMEOUT
, or 
UNEXPECTED
. * The "ErrMsg" Column: This column will tell you exactly why the clone failed (e.g., "remote copy failed", "disk full", or "timeout"). Step B: Check BE Logs Log into your BE nodes and search for clone-related errors in the 
<http://be.INFO|be.INFO>
log:
Copy code
bash
grep "clone" /opt/starrocks/be/log/be.INFO | grep -i "fail"
This will often show lower-level details like "Connection refused" or specific filesystem errors. Step C: Check for "Bad" Replicas If the cluster is repeatedly trying to repair a specific tablet and failing, you can find unhealthy replicas with:
Copy code
sql
ADMIN SHOW REPLICA STATUS FROM <table_name> WHERE STATUS != "OK";
3. Quick Fixes * If Disks are Full: Clean up disk space or add more disks/nodes. * If a BE is Unstable: Restart the BE node that is failing to receive or send clones. * If the Alert is Transient: Sometimes heavy loading tasks cause clone timeouts. If the alert clears itself and 
history_tablets
shows recent successes, it may have been a temporary resource spike. * Manual Repair: If a specific tablet is stuck, you can tell the cluster the replica is "bad" to force a fresh clone:
Copy code
sql
    ADMIN SET REPLICA STATUS PROPERTIES("tablet_id" = "12345", "backend_id" = "10001", "status" = "bad");
Summary Recommendation Check 
SHOW PROC '/cluster_balance/history_tablets';
 first. It is the single most useful tool for identifying the failing node and the specific error message. References • administration/management/resource_management/Replica.md
h
@Rocky Found this after running SHOW PROC '/cluster_balance/history_tablets';
Copy code
Type: REPAIR
        Medium: HDD
        Status: VERSION_INCOMPLETE
         State: FINISHED
r
The output you see means that StarRocks attempted a background repair of a tablet, but even though the repair task "finished" its execution, the tablet is still not fully healthy because it is missing data versions. Here is the breakdown of what those specific fields mean: * Type: REPAIR: The FE (Frontend) detected a replica was inconsistent or lagging and automatically scheduled a task to fix it. * State: FINISHED: The specific clone task assigned to a Backend (BE) has completed its run. It does not necessarily mean the tablet is now healthy; it just means the worker process finished copying what it was told to copy. * Status: VERSION_INCOMPLETE: This is the core issue. It indicates that the replica on the destination BE is still missing some data versions compared to the healthy replicas. This often happens because: * The source replica (where it was cloning from) underwent "compaction" and deleted the older versions the destination node still needed. * There was a high frequency of writes (Stream Load/Routine Load) occurring during the repair, and the replica couldn't catch up fast enough. * (In some versions) A bug or a missing segment file on the source node prevents the version from ever being fully "complete." How to Fix and Investigate 1. Identify the Tablet Find the 
TabletId
from the same row in the 
SHOW PROC '/cluster_balance/history_tablets';
output. 2. Check the Replica Health Run the following to see the version status of every copy of that tablet:
Copy code
sql
SHOW TABLET <TabletId>;
Look at the DetailCmd column in the result. Copy and run that command (it will look like 
SHOW PROC '/dbs/123/456/...'
). * Compare the Version column for all replicas. You will likely see one node has a much lower version number than the others. 3. Force a Full Repair If the automatic REPAIR keeps finishing with 
VERSION_INCOMPLETE
, the incremental sync is stuck. You can force StarRocks to perform a Full Clone (deleting the old data and copying everything fresh) by marking the lagging replica as "bad":
Copy code
sql
-- Replace <TabletId> and <BackendId> with the values from the previous step
ADMIN SET REPLICA STATUS PROPERTIES("tablet_id" = "<TabletId>", "backend_id" = "<BackendId>", "status" = "bad");
Once marked as "bad," the FE will notice the replica is effectively "missing" and will perform a full clone from a healthy node. 4. Check BE Logs for Specific Errors If the tablet stays in this state even after a manual repair, check the BE logs (
<http://be.INFO|be.INFO>
) on the node that is the destination of the clone. Search for the tablet ID:
Copy code
bash
grep "<TabletId>" /opt/starrocks/be/log/be.INFO | grep -iE "clone|snapshot"
Look for errors like: * `no version to clone`: The source node has already cleaned up the data you need. * `get_rowsets_for_snapshot failed`: There is a physical file missing or corrupted on the source. Summary The repair task "finished" but failed to reach the target version. In most cases, marking the specific replica as "bad" is the quickest way to force a clean, successful sync and resolve the alert. References • faq/loading/Loading_faq.md • faq/operation_maintenance_faq.md • administration/management/resource_management/Replica.md • administration/management/FE_configuration.md
h
@Rocky Why create materialized view (asynchronous) default "storage_medium" to "HDD"? Would the performance drop if it is set to HDD? What are the scenarios that materialized view is preferred over logical view?
r
Why does 
storage_medium
default to "HDD"? In StarRocks, the 
storage_medium
property for asynchronous materialized views (and regular tables) follows a specific inference logic based on your Backend (BE) disk configuration: 1. Automatic Inference: If you do not explicitly specify 
storage_medium
, StarRocks looks at the disks reported by your BEs (
storage_root_path
). 2. Hybrid Disks: If your BEs have both SSD and HDD disks, StarRocks defaults to HDD unless you also specify a 
storage_cooldown_time
(in which case it starts on SSD and migrates to HDD later). 3. Single Disk Type: If your BEs only have HDDs, it defaults to HDD. If they only have SSDs, it defaults to SSD. The system defaults to HDD in hybrid environments to prevent accidentally consuming expensive SSD space for all data by default. ────────── Performance Impact of HDD Yes, there is a performance difference, but its impact depends on your workload and caching: * I/O Latency: HDDs have significantly higher seek times and lower throughput than SSDs. If your Materialized View (MV) is large and the query must read from disk (cold data), you will see slower query response times compared to SSD. * Refresh Speed: Since asynchronous MVs must "write" results back to disk during a refresh, an HDD-backed MV will take longer to refresh than an SSD-backed one. * Mitigation (Page Cache): StarRocks enables a Page Cache by default (typically 20% of memory). If your MV's "hot" data fits in memory, the performance difference between HDD and SSD becomes negligible for repeat queries. * Recommendation: For high-concurrency, low-latency dashboard queries, it is recommended to set 
"storage_medium" = "SSD"
. ────────── Materialized View vs. Logical View: When to use which? Feature | ***Logical View*** | ***Materialized View (Async)*** Storage | No physical storage. | Physically stores pre-computed results. Performance | No speedup (runs the full SQL every time). | High speedup (queries the pre-computed table). Data Freshness | Always real-time. | Eventually consistent (depends on refresh interval). Resource Usage | High CPU/Memory during query time. | CPU/Memory used during **refresh**; low usage during query. Query Rewrite | No automatic rewrite. | Supports Transparent Rewrite (automatically speeds up existing queries). Use a Logical View when: * You only need to simplify complex SQL or wrap business logic (e.g., masking columns). * The underlying data changes so frequently that a refresh would be constant. * The query is already fast enough and you don't want to pay for extra storage. Use an Asynchronous Materialized View when: * Query Acceleration: Your queries involve heavy joins or aggregations that take seconds/minutes to run. * Data Lake Integration: You want to accelerate queries against external data (Hive, Hudi, Iceberg) by caching results locally in StarRocks. * Transparent Optimization: You want to speed up existing reports without changing the application's SQL (via Query Rewrite). * Pre-aggregation: You are building a "Metric Store" or "Gold Layer" where data is rolled up (e.g., from raw logs to daily summaries). References • using_starrocks/async_mv/use_cases/data_modeling_with_materialized_views.md • sql-reference/sql-statements/table_bucket_part_index/CREATE_TABLE.md • sql-reference/sql-statements/materialized_view/CREATE_MATERIALIZED_VIEW.md
h
@Rocky Why I can't find storage_medium when running show create table? And the BE uses SSD drive but show materialized view shows storage_medium is HDD
r
The reason you see HDD even though your hardware is an SSD is likely because the disk was not explicitly labeled as an SSD in the Backend (BE) configuration. StarRocks does not always "auto-detect" the physical hardware type; it relies on how the paths are defined in your configuration files. 1. Why 
storage_medium
shows as HDD when the disk is SSD In StarRocks, the storage medium is determined by the 
storage_root_path
setting in each BE's 
be.conf
file. * Explicit Labeling: If you do not add 
,medium:ssd
to your path, StarRocks may default to treating it as HDD. * Default Behavior: If your 
be.conf
looks like this: 
storage_root_path = /data/storage
, StarRocks often defaults this to HDD. * To Fix This: You must update the 
be.conf
on every BE node to explicitly tell StarRocks it is an SSD:
Copy code
bash
    # Example be.conf change
    storage_root_path = /data/storage,medium:ssd
Note: After changing this, you must restart the BE. 2. Why you can't find 
storage_medium
in 
SHOW CREATE TABLE
 
SHOW CREATE TABLE
typically only displays properties that are explicitly set or deviate from the system default. * Inferred Defaults: If you didn't specify 
storage_medium
during creation and the system defaulted it (based on your 
be.conf
), the property is often hidden in the "Create Table" output because it is considered a default state. * Where to check for sure: Instead of 
SHOW CREATE TABLE
, you can check the actual metadata for each partition to see what medium is being used:
Copy code
sql
    SHOW PARTITIONS FROM <table_name>;
Look for the StorageMedium column in the results. This will show you exactly what the FE (Frontend) has assigned to that specific piece of data. 3. How to verify what the FE thinks about your disks To see how the FE perceives your Backend storage, run:
Copy code
sql
SHOW BACKENDS\G
In the 
Status
column (which is a JSON string), you can often see the disk information. For more detail, you can use the internal process view:
Copy code
sql
SHOW PROC '/backends';
Then find the 
BackendId
and run:
Copy code
sql
-- Replace <BackendId> with the actual ID
SHOW PROC '/backends/<BackendId>';
This will list each disk path on that BE and clearly show whether StarRocks thinks it is HDD or SSD. Summary of Action Items: 1. Check 
SHOW PROC '/backends/<id>'
to see the current detected medium. 2. If it says HDD but you have SSDs, update 
be.conf
with 
,medium:ssd
in the 
storage_root_path
. 3. Restart the BEs. 4. For existing tables/MVs, you may need to manually alter them if they were already created with the wrong medium:
Copy code
sql
    ALTER TABLE <name> MODIFY PARTITION (*) SET ("storage_medium" = "SSD");
References • administration/management/BE_configuration.md • sql-reference/sql-statements/table_bucket_part_index/CREATE_TABLE.md • sql-reference/sql-statements/table_bucket_part_index/SHOW_PARTITIONS.md References • administration/management/FE_configuration.md • deployment/environment_configurations.md • deployment/plan_cluster.md • administration/management/BE_configuration.md • faq/Others.md • sql-reference/sql-statements/cluster-management/nodes_processes/SHOW_BACKENDS.md
h
@Rocky What would happen if storage_medium is set to HDD but the BE disk is actually SSD?
r
If you have physical SSDs but the 
storage_medium
is logically set to HDD in StarRocks, the impact is primarily on data placement logic and lifecycle management, rather than a direct software-imposed performance throttle. Here is what happens in that scenario: 1. Physical Performance vs. Logical Label * No "Artificial" Slowdown: StarRocks does not artificially slow down I/O just because a disk is labeled "HDD." Your physical SSD will still provide high IOPS and low latency. * OS Scheduling: As long as your operating system is configured with the correct scheduler (like 
none
or 
kyber
for SSDs), the hardware will perform at its native speed. 2. Impact on Data Placement & Rebalancing The 
storage_medium
property is the "hint" that the Frontend (FE) uses to decide where to put your data. * Priority: If your cluster has a mix of real HDDs and real SSDs (properly labeled in 
be.conf
), and you set a table to 
HDD
, the FE will avoid placing that table's data on your SSDs to save the "fast" space for other tables. * Imbalance: If all your disks are SSDs but are labeled as HDD in the configuration, the FE will treat them all as a single tier of storage. You won't see a performance drop, but you lose the ability to use the "Storage Cooldown" feature. 3. Loss of "Storage Cooldown" (Tiered Storage) This is the biggest functional loss. StarRocks has a feature where data starts on SSD (for fast ingestion/queries) and automatically migrates to HDD after a certain time (to save costs). * Cooldown Requirement: This feature only works if the initial 
storage_medium
is set to 
SSD
. * The Result: If your table is set to 
HDD
from the start, StarRocks assumes it is already on the "cold/cheap" tier. It will never trigger a migration task, even if you configure 
storage_cooldown_ttl
. 4. Background Task Priority StarRocks uses the medium label to prioritize certain background operations: * Migration Threads: There are specific BE configurations (like 
storage_medium_migrate_count
) that control how many threads are used to move data between different mediums. * Compaction: While compaction happens on both, the system's internal heuristics for managing I/O pressure are sometimes more aggressive on SSDs because they can handle higher concurrency. Labeling an SSD as an HDD might result in slightly more conservative background resource allocation in very high-load scenarios. Summary: Should you fix it? If your hardware is all SSD, you should ideally fix the labels to reflect reality. Why fix it? 1. Future-Proofing: If you ever add actual HDDs later, your data placement logic will already be correct. 2. Tiering: It allows you to use 
storage_cooldown_ttl
to manage your data lifecycle. 3. Accuracy: 
SHOW PROC '/backends/'
and other monitoring tools will give you a truthful view of your cluster's resource utilization. How to fix it: 1. Update 
be.conf
for each BE: 
storage_root_path = /your/path,medium:ssd
. 2. Restart the BEs. 3. Update existing tables: 
ALTER TABLE <name> MODIFY PARTITION (*) SET ("storage_medium" = "SSD");
. References • sql-reference/sql-statements/table_bucket_part_index/CREATE_TABLE.md • administration/management/FE_configuration.md • faq/operation_maintenance_faq.md • table_design/table_design.md • faq/Others.md • introduction/Architecture.md
h
@Rocky To alter the storage_medium from HDD to SSD, what if there are many partitions? Would it cause performance issues when the partitions or materialized views are being altered?
r
Altering the 
storage_medium
for many partitions is a safe operation in StarRocks. It is designed to be a background task that does not block your data's availability. Here is a detailed breakdown of what happens when you run `ALTER TABLE ... MODIFY PARTITION (*)`: 1. Is there an immediate performance drop? No. The 
ALTER
command itself is a metadata-only operation in the Frontend (FE). * No Heavy Locking: It does not take a long-term "exclusive lock" on the table. Your queries and data loading will continue to run normally. * Metadata Change: The FE simply updates the "desired" storage medium for the tablets in its catalog and then returns "OK" to you almost instantly. 2. How the background migration works Once the metadata is updated, the Frontend instructs the Backends (BEs) to begin migrating the physical data (tablets) from HDD to SSD. * IO & CPU Usage: The migration involves reading data from the HDD and writing it to the SSD. This will consume some disk I/O and a small amount of CPU on the BE nodes. * Throttling: To prevent this from slowing down your active queries, StarRocks limits how many migrations happen at once. * Configuration: The speed is controlled by the BE parameter 
storage_medium_migrate_count
(default is 3 threads). This means each BE will only move 3 tablets simultaneously, ensuring that background movement doesn't overwhelm the system. 3. Special Considerations for Materialized Views Since an asynchronous Materialized View (MV) is physically stored as a table, you use the same command to alter it:
Copy code
sql
-- Use the MV name just like a table name
ALTER TABLE your_mv_name MODIFY PARTITION (*) SET ("storage_medium" = "SSD");
* Refresh Tasks: The migration happens independently of the MV refresh process. If an MV refresh triggers while a migration is in progress, the new data generated by the refresh will be written directly to the SSD (the new target), while the old data continues to migrate in the background. * No Integrity Risk: StarRocks ensures data consistency throughout the move. If a query hits a tablet that is currently being moved, it will simply read from the original (HDD) location until the migration of that specific tablet is 100% complete. 4. How to monitor the progress You can track how many partitions have successfully moved to the SSD by running:
Copy code
sql
SHOW PARTITIONS FROM your_table_or_mv_name;
Look at the StorageMedium column. It will update from HDD to SSD as each partition finishes its migration. Summary Checklist * Safety: High. It's a standard background operation. * Impact: Low. I/O is throttled by 
storage_medium_migrate_count
. * Benefit: Once a tablet is moved to SSD, queries hitting that data will immediately start seeing the lower latency and higher throughput of the SSD hardware. References • sql-reference/sql-statements/table_bucket_part_index/ALTER_TABLE.md • administration/management/BE_configuration.md • faq/operation_maintenance_faq.md References • faq/operation_maintenance_faq.md • using_starrocks/async_mv/use_cases/create_partitioned_materialized_view.md • sql-reference/sql-statements/materialized_view/ALTER_MATERIALIZED_VIEW.md • sql-reference/sql-statements/table_bucket_part_index/ALTER_TABLE.md
h
@Rocky To alter the storage_medium from HDD to SSD, what if there are many partitions? Would it cause performance issues when the partitions or materialized views are being altered? Please note that no new disk is being introduced. The purpose of the alter is simply because of wrong configuration to make starrocks to default the medium to HDD even though the disk is SSD
r
If your physical disks are already SSDs and you have corrected your 
be.conf
to reflect this, altering the 
storage_medium
property for your partitions is extremely safe and will not cause performance issues. Here is exactly what happens in your specific scenario: 1. It becomes a "No-Op" (Metadata-only) change When you run 
ALTER TABLE ... MODIFY PARTITION (*) SET ("storage_medium" = "SSD")
, the StarRocks Frontend (FE) checks the current location of every tablet. * The check: The FE looks at the disk where a tablet is stored and asks: "Is this disk an SSD?" * The result: Since you have already updated your 
be.conf
(adding 
,medium:ssd
) and restarted the BEs, the BEs are now reporting those disks as SSD to the FE. * The behavior: Because the tablets are already sitting on disks that the system now recognizes as SSD, no physical data migration will occur. The FE simply updates the metadata for those partitions to match the reality of the hardware. This is nearly instantaneous. 2. Why there is no performance impact * No Data Movement: Since the tablets don't need to move to a different disk, there is no disk I/O or network traffic generated by "migration." * FE Metadata Update: Updating the label for thousands of partitions is a lightweight transaction in the FE's image/journal. It does not lock the table for queries or loads. * Query Consistency: Your queries will continue to run against the same physical data. The only change is that StarRocks now "knows" it's on fast storage, which allows features like Storage Cooldown to be enabled in the future. 3. Recommended Order of Operations To ensure this goes smoothly without any "Tablet Migrate Failed" errors, follow this order: 1. Update 
be.conf
first: Add 
,medium:ssd
to your 
storage_root_path
on all BE nodes. 2. Restart BEs: One by one (rolling restart) so the FE receives the updated disk reports. 3. Verify via PROC: Run 
SHOW PROC '/backends';
and click into a backend to verify that the 
StorageMedium
for the paths now shows SSD. 4. Run the Alter: Execute the 
ALTER TABLE ... MODIFY PARTITION (*)
command. 4. What about Materialized Views? The same logic applies. If the underlying data for the MV is already on the disks that are now labeled SSD, the alter command will simply update the MV's metadata without moving a single byte of data. Summary Because you are not introducing new disks and are simply correcting the label of the existing disks, the "migration" is a logical one. You can safely run this on tables with a large number of partitions without worrying about resource contention or downtime. References • sql-reference/sql-statements/table_bucket_part_index/ALTER_TABLE.md • administration/management/BE_configuration.md • faq/operation_maintenance_faq.md • sql-reference/sql-statements/table_bucket_part_index/CREATE_TABLE.md References • faq/operation_maintenance_faq.md • administration/management/FE_configuration.md • table_design/table_design.md • sql-reference/sql-statements/table_bucket_part_index/CREATE_TABLE.md • administration/management/resource_management/spill_to_disk.md
h
@Rocky How to find out what users belong to a specific role?
r
To find out which users belong to a specific role in StarRocks, the most direct method is to query the 
sys.role_edges
 system view. Using 
sys.role_edges
 This view displays the relationships between roles and their grantees (either other roles or users). To find all users assigned to a role named 
'your_role_name'
, run:
Copy code
sql
SELECT TO_USER 
FROM sys.role_edges 
WHERE FROM_ROLE = 'your_role_name' 
  AND TO_USER IS NOT NULL;
How to read the results: * 
FROM_ROLE
: The role you are interested in. * 
TO_USER
: The user to whom that role is granted. * 
TO_ROLE
: If this column is populated instead of 
TO_USER
, it means the role was granted to another role (role inheritance). Important Requirements * Privileges: By default, you must have the 
user_admin
 system role to query this view. * Availability: This view is available in StarRocks versions 3.0 and later. Alternative: Checking a Specific User If you want to go the other way and check what roles a specific user has, you can use:
Copy code
sql
SHOW GRANTS FOR 'username'@'host';
References • sql-reference/sys/role_edges.md • administration/user_privs/authorization/User_privilege.md • sql-reference/sql-statements/account-management/SHOW_ROLES.md References • sql-reference/sys/role_edges.md • administration/user_privs/authorization/User_privilege.md • sql-reference/sql-statements/account-management/SHOW_ROLES.md • administration/user_privs/authorization/privilege_faq.md
h
@Rocky How to lock a user?
r
In StarRocks, there is no explicit 
ACCOUNT LOCK
command like in some other databases (e.g., Oracle or standard MySQL). To achieve the same effect and prevent a user from logging in without deleting the account, you should use the following method: 1. Set Maximum Connections to 0 The most effective way to "lock" a user in StarRocks is to set their allowable connections to zero. This prevents any new sessions from being established for that user.
Copy code
sql
-- Lock the user by setting max connections to 0
ALTER USER 'username' SET PROPERTIES ("max_user_connections" = "0");
2. How to "Unlock" the User To allow the user to log in again, simply set the property back to a positive integer (the default is often 100 or higher depending on your cluster setup).
Copy code
sql
-- Unlock the user by restoring connection limits
ALTER USER 'username' SET PROPERTIES ("max_user_connections" = "1000");
3. Alternative: Drop the User If the user account is no longer needed at all, you can permanently remove it:
Copy code
sql
DROP USER 'username'@'host';
Important Notes: * Privileges: You must have the 
user_admin
system role to perform these actions on other users. * Metadata Only: Setting 
max_user_connections
is a metadata change and takes effect immediately for all new connection attempts. * Active Sessions: Locking a user via 
max_user_connections = 0
prevents new logins but may not automatically kill an existing active session. To kill an existing session, you can use:
Copy code
sql
    SHOW PROCESSLIST;
    KILL [connection_id];
References • sql-reference/sql-statements/account-management/ALTER_USER.md • administration/user_privs/authentication/native_authentication.md • faq/operation_maintenance_faq.md References • administration/user_privs/authentication/native_authentication.md • deployment/post_deployment_setup.md • quick_start/helm.md • administration/user_privs/ssl_authentication.md
h
@Rocky When creating a role and the role only needs to access the default_catalog, is it required to
Copy code
GRANT USAGE ON CATALOG default_catalog TO ROLE
r
No, it is not strictly required to explicitly grant 
USAGE
on the 
default_catalog
for a role to access internal data, but it is highly recommended to avoid friction with certain commands. Here is the breakdown of how the 
default_catalog
behaves regarding the 
USAGE
privilege: 1. Implicit Access for Queries By default, when a user connects to StarRocks (via MySQL client or JDBC), they are placed inside the 
default_catalog
automatically. * If a role has privileges on a specific database or table (e.g., 
GRANT SELECT ON ALL TABLES IN DATABASE my_db TO ROLE my_role
), the user can query those objects without having an explicit 
USAGE
grant on the 
default_catalog
. * Because you are already "inside" the internal catalog upon login, the system does not strictly block data access for lack of the catalog-level 
USAGE
privilege. 2. When 
USAGE
is Required Even for the 
default_catalog
, the 
USAGE
privilege is required if the user needs to: * Run 
SHOW CATALOGS
: Without 
USAGE
, the 
default_catalog
will not appear in the list. * Run 
SET CATALOG default_catalog
: If the user has switched to an external catalog (like a Hive or Iceberg catalog) and wants to switch back to the internal one, the 
SET CATALOG
command requires 
USAGE
on the target catalog. * Run 
SHOW DATABASES FROM default_catalog
: This explicit metadata query requires catalog-level usage. 3. Best Practice To ensure a consistent experience (especially if your users might switch between catalogs), it is standard practice to grant 
USAGE
on the 
default_catalog
to the 
public
role. Since every user in StarRocks automatically inherits the 
public
role, this ensures everyone can see the internal catalog and switch back to it if they leave.
Copy code
sql
-- This ensures all users can always see and switch back to the internal catalog
GRANT USAGE ON CATALOG default_catalog TO ROLE public;
Summary: For basic 
SELECT
queries on internal tables where the user is already in the default context, you don't need it. For a fully functional role that can navigate the system, you should grant it. References • data_source/catalog/default_catalog.md • administration/user_privs/authorization/privilege_item.md • sql-reference/sql-statements/account-management/GRANT.md • administration/user_privs/authorization/built_in_roles.md • data_source/catalog/iceberg/iceberg_rest_security.md References • administration/user_privs/authorization/privilege_item.md • data_source/catalog/default_catalog.md • data_source/catalog/iceberg/iceberg_rest_security.md • administration/user_privs/authorization/built_in_roles.md • administration/user_privs/authorization/user_privs.md
h
@Rocky When creating a role and granting the permissions to objects within the default_catalog, is it required to SET CATALOG default_catalog; before granting the permissions?
r
Yes, you must ensure the session is set to the correct catalog before granting permissions on databases or tables. This is because the 
GRANT
syntax for those objects in StarRocks does not currently support fully qualified names that include the catalog (e.g., you cannot do 
GRANT SELECT ON TABLE default_catalog.db.tbl
). Here is the breakdown of how it works: 1. The Context Matters The 
GRANT
command for databases and tables operates on the current catalog of your session. _ If you just logged in, your session is automatically set to 
default_catalog
by default. In this case, you do not _need* to run 
SET CATALOG default_catalog;
because you are already there. * However, if you have previously executed 
SET CATALOG some_external_catalog;
in your current session, any subsequent 
GRANT ON DATABASE ...
or 
GRANT ON TABLE ...
will attempt to apply to that external catalog. 2. When you MUST use 
SET CATALOG
 You must run 
SET CATALOG
if: * You want to grant permissions on objects in a catalog other than the one you are currently in. * You are writing an automation script and want to be 100% sure the permissions are applied to the correct catalog regardless of the connection's initial state. 3. Example Workflow To grant permissions on a database inside the internal catalog:
Copy code
sql
-- Ensure you are in the internal catalog
SET CATALOG default_catalog;

-- Grant permissions on the database
GRANT ALL PRIVILEGES ON DATABASE my_internal_db TO ROLE my_role;

-- Grant permissions on all tables in that database
GRANT SELECT ON ALL TABLES IN DATABASE my_internal_db TO ROLE my_role;
Exceptions * Catalog-level grants: Commands like 
GRANT USAGE ON CATALOG default_catalog TO ROLE my_role;
do not require you to be in that catalog first, as the catalog name is explicitly part of the syntax. * System-level grants: Commands like 
GRANT CREATE RESOURCE ON SYSTEM TO ROLE my_role;
are global and do not depend on the catalog context. Summary: While 
default_catalog
is the default upon login, it is a best practice to include 
SET CATALOG default_catalog;
in setup scripts to ensure the 
GRANT
statements target the correct metadata layer. References • sql-reference/sql-statements/account-management/GRANT.md • data_source/catalog/default_catalog.md • administration/user_privs/authorization/privilege_item.md References • sql-reference/sql-statements/account-management/GRANT.md
h
@Rocky Would this work for Starrocks?
Copy code
ALTER USER 'new_user'@'%' REQUIRE SSL;
r
No, the command 
ALTER USER 'new_user'@'%' REQUIRE SSL;
will not work in StarRocks. While StarRocks is compatible with the MySQL wire protocol, it does not currently support the per-user 
REQUIRE SSL
syntax in 
CREATE USER
or 
ALTER USER
statements. How to Force SSL in StarRocks StarRocks (v3.4.1 and later) manages SSL enforcement at the cluster level rather than the user level. To force all users to use SSL, you must configure the Front End (FE) nodes via the 
fe.conf
file: 1. Enable SSL by providing your keystore details in 
fe.conf
. 2. Force SSL by setting the following parameter:
Copy code
properties
    ssl_force_secure_transport = true
When this is set to 
true
, the FE will reject any connection that is not encrypted with SSL. Summary of SSL support * Availability: Introduced in version 3.4.1. * Global Enforcement: Controlled by 
ssl_force_secure_transport
in 
fe.conf
. * Per-User Enforcement: Not currently supported. You cannot selectively require SSL for some users while allowing cleartext for others. If you attempt to run 
ALTER USER ... REQUIRE SSL
, you will likely receive a syntax error because the parser does not recognize the 
REQUIRE
clause for the 
ALTER USER
statement. References • administration/user_privs/ssl_authentication.md • sql-reference/sql-statements/account-management/ALTER_USER.md • sql-reference/sql-statements/account-management/CREATE_USER.md References • administration/user_privs/ssl_authentication.md • sql-reference/sql-statements/account-management/CREATE_USER.md
h
@Rocky Would this work?
Copy code
SUBMIT TASK my_hourly_task
SCHEDULE EVERY 1 HOUR
AS
TRUNCATE TABLE target_table;
INSERT INTO target_table SELECT * FROM source_table;
r
No, that exact syntax will not work in StarRocks for two reasons: 1. Multiple Statements: 
SUBMIT TASK
currently only supports a single ETL statement (like 
INSERT
or 
CREATE TABLE AS SELECT
). It cannot execute multiple statements separated by semicolons. 2. Schedule Syntax: The 
SCHEDULE
clause requires the 
INTERVAL
keyword and parentheses. Recommended Solution In StarRocks, the standard way to perform a "truncate and reload" operation in a single atomic step is using 
INSERT OVERWRITE
. This replaces the data in the target table with the results of the query without needing a separate 
TRUNCATE
command. Corrected Query:
Copy code
sql
SUBMIT TASK my_hourly_task
SCHEDULE EVERY(INTERVAL 1 HOUR)
AS
INSERT OVERWRITE target_table 
SELECT * FROM source_table;
Key Differences: * 
INSERT OVERWRITE
: This is the best practice for refreshing a table. It is more efficient than 
TRUNCATE
+ 
INSERT
because it handles the data swap atomically. * Schedule Clause: Must be 
EVERY(INTERVAL 1 HOUR)
. * Atomicity: 
INSERT OVERWRITE
ensures that the table is never empty for users querying it during the reload; the old data is only replaced once the new data is fully prepared. Monitoring your task: After submitting, you can check the status and history of your task using these system views:
Copy code
sql
-- View task definition
SELECT * FROM INFORMATION_SCHEMA.tasks WHERE task_name = 'my_hourly_task';

-- View execution history
SELECT * FROM INFORMATION_SCHEMA.task_runs WHERE task_name = 'my_hourly_task';
References • sql-reference/sql-statements/loading_unloading/ETL/SUBMIT_TASK.md • loading/InsertInto.md • sql-reference/sql-statements/loading_unloading/INSERT.md References • sql-reference/sql-statements/loading_unloading/ETL/SUBMIT_TASK.md • loading/InsertInto.md
h
@Rocky What permission is required to run SHOW GRANTS FOR?
r
The permissions required for 
SHOW GRANTS
depend on whose permissions you are trying to view: 1. To view your own permissions (
SHOW GRANTS;
): No special permissions are required. Every user is allowed to see their own granted privileges and assigned roles. 2. To view permissions for another user or role (
SHOW GRANTS FOR <user_or_role>;
): You must have the 
user_admin
 system-defined role. Only users with this role (or the 
db_admin
/ 
root
which inherit high-level privileges) can inspect the privileges of other identities in the cluster. Summary Table Statement | Required Permission/Role 
SHOW GRANTS;
| None (any user can run this) 
SHOW GRANTS FOR 'user'@'%';
| 
user_admin
role 
SHOW GRANTS FOR ROLE role_name;
| 
user_admin
role Alternative: System Views If you are an administrator and want to query permissions across the entire system using SQL filters, you can also query the following system views (which also require the 
user_admin
role by default): * 
sys.grants_to_users
* 
sys.grants_to_roles
References • sql-reference/sql-statements/account-management/SHOW_GRANTS.md • administration/user_privs/authorization/User_privilege.md • sql-reference/sys/grants_to_users.md References • sql-reference/sql-statements/account-management/SHOW_GRANTS.md • administration/user_privs/authorization/User_privilege.md • sql-reference/sys/grants_to_users.md • sql-reference/sys/grants_to_roles.md References • sql-reference/sql-statements/account-management/SHOW_GRANTS.md • sql-reference/sys/grants_to_users.md • sql-reference/sys/grants_to_roles.md • sql-reference/information_schema/schema_privileges.md • administration/user_privs/authorization/User_privilege.md
h
@Rocky Show me the queries to list grantee, role, and permissions using these two views
Copy code
*   sys.grants_to_users
*   sys.grants_to_roles
r
You can use the following queries to inspect privileges. Note that 
grants_to_users
shows privileges granted directly to users, while 
grants_to_roles
shows privileges granted to roles. 1. List privileges granted directly to Users This query shows which users have been given specific permissions directly (without a role).
Copy code
sql
SELECT 
    GRANTEE, 
    OBJECT_TYPE, 
    OBJECT_NAME, 
    PRIVILEGE_TYPE, 
    IS_GRANTABLE 
FROM sys.grants_to_users;
2. List privileges granted to Roles This query shows the permissions assigned to each user-defined role.
Copy code
sql
SELECT 
    GRANTEE AS ROLE_NAME, 
    OBJECT_TYPE, 
    OBJECT_NAME, 
    PRIVILEGE_TYPE, 
    IS_GRANTABLE 
FROM sys.grants_to_roles;
3. Bonus: See which Users have which Roles If you want to see the "Grantee → Role" mapping to understand which users inherit the permissions from the query above, use the 
role_edges
view:
Copy code
sql
SELECT 
    GRANTEE, 
    ROLE 
FROM sys.role_edges;
Key Column Descriptions: * GRANTEE: The user (e.g., 
'rachel'@'%'
) or the role name receiving the permission. * OBJECT_TYPE: The level of the privilege (e.g., 
DATABASE
, 
TABLE
, 
SYSTEM
). * OBJECT_NAME: The specific name of the database or table. * PRIVILEGE_TYPE: The actual action allowed (e.g., 
SELECT
, 
INSERT
, 
CREATE TABLE
). Note: Accessing these views requires the 
user_admin
role. References • sql-reference/sys/grants_to_users.md • sql-reference/sys/grants_to_roles.md • sql-reference/sys/role_edges.md References • sql-reference/sys/grants_to_users.md • sql-reference/sys/grants_to_roles.md • sql-reference/information_schema/column_privileges.md • sql-reference/sys/sys.md • sql-reference/sys/role_edges.md
h
@Rocky Please show me one query to list grantee, role, and permissions by joining
Copy code
sys.grants_to_users
sys.grants_to_roles
sys.role_edges
r
To list all permissions—including those granted directly to users and those inherited through roles—you can use a 
UNION ALL
to combine the data from these three views into a single result set. The following query identifies the Grantee (the user), whether the permission is Direct or comes from a Role, and the specific Object and Privilege details:
Copy code
sql
-- 1. Get privileges granted directly to users
SELECT 
    GRANTEE, 
    'Direct' AS GRANTED_VIA, 
    OBJECT_TYPE, 
    CONCAT_WS('.', OBJECT_CATALOG, OBJECT_DATABASE, OBJECT_NAME) AS OBJECT,
    PRIVILEGE_TYPE
FROM sys.grants_to_users

UNION ALL

-- 2. Get privileges granted to users via roles
SELECT 
    re.TO_USER AS GRANTEE, 
    re.FROM_ROLE AS GRANTED_VIA, 
    gr.OBJECT_TYPE, 
    CONCAT_WS('.', gr.OBJECT_CATALOG, gr.OBJECT_DATABASE, gr.OBJECT_NAME) AS OBJECT,
    gr.PRIVILEGE_TYPE
FROM sys.role_edges re
JOIN sys.grants_to_roles gr ON re.FROM_ROLE = gr.GRANTEE
WHERE re.TO_USER IS NOT NULL;
Explanation: * 
sys.grants_to_users
: Provides privileges explicitly assigned to a user account. * 
sys.role_edges
: Acts as the bridge, mapping which users (
TO_USER
) belong to which roles (
FROM_ROLE
). * 
sys.grants_to_roles
: Provides the actual permissions assigned to those roles. * 
CONCAT_WS
: Used here to create a readable string of the object path (Catalog.Database.Table). Note on Nested Roles: If your RBAC structure uses nested roles (roles granted to other roles), this query will only show permissions for roles granted directly to users. To see the full inheritance tree for deeply nested roles, you would typically need a recursive query or to inspect the 
role_edges
view repeatedly. References • sql-reference/sys/grants_to_users.md • sql-reference/sys/grants_to_roles.md • sql-reference/sys/role_edges.md References • sql-reference/sys/role_edges.md • sql-reference/sys/grants_to_roles.md • sql-reference/sys/grants_to_users.md • sql-reference/sys/sys.md • sql-reference/information_schema/schema_privileges.md
h
@Rocky I got this error
Copy code
sqlalchemy.exc.ProgrammingError: (pymysql.err.ProgrammingError) (1064, 'StarRocks planner use long time 3204 ms in logical phase, This probably because 1. FE Full GC, 2. Hive external table fetch metadata took a long time, 3. The SQL is very complex. You could 1. adjust FE JVM config, 2. try query again, 3. enlarge new_planner_optimize_timeout session variable')
r
The error 
StarRocks planner use long time ... in logical phase
(Error 1064) occurs when the StarRocks query optimizer (the "planner") exceeds its allotted time limit (default 3000ms) while trying to generate an execution plan. This is usually caused by one of three reasons: 1. Complex SQL: The query has a large number of joins or subqueries that require more time to optimize. 2. FE Full GC: The Frontend (FE) is experiencing a "Stop-the-World" Garbage Collection, pausing the planner's work. 3. Slow Metadata Access: For external tables (like Hive or Iceberg), fetching metadata from the remote metastore is taking too long. Recommended Solutions 1. Increase the Optimizer Timeout (Quick Fix) If the query is complex but valid, you can increase the timeout for the current session or globally. * Session Level:
Copy code
sql
    SET new_planner_optimize_timeout = 10000; -- Set to 10 seconds
* Global Level:
Copy code
sql
    SET GLOBAL new_planner_optimize_timeout = 10000;
2. Check and Adjust FE Memory (Long-term Fix) If this happens frequently or for simple queries, it is likely a Full GC issue. * Diagnose: Check the 
fe.gc.log
(located in the FE 
log/
directory) or run 
jstat -gcutil <FE_PID> 1000
to see if the Old Generation memory (
O
) is near 100%. * Fix: Increase the JVM heap size in 
fe/conf/fe.conf
. Locate the 
JAVA_OPTS
line and increase 
-Xmx
and 
-Xms
. _ _Example:* For clusters with ~1 million tablets, at least 16GB is recommended (
-Xmx16g -Xms16g
). _ _Rule of thumb:* If your FE has more than 32GB of RAM, you can set 
-Xmx
to 80% of total memory. 3. Optimize External Metadata (If applicable) If you are querying Hive/Iceberg/JDBC tables: * Ensure the connection to the external Metastore is stable. * Check if you can enable or refresh the Metadata Cache for the specific catalog to speed up planning. Summary of System Variables Variable | Default | Unit | Description 
new_planner_optimize_timeout
| 3000 | ms | Max time the optimizer can spend planning a query. 
query_timeout
| 300 | sec | Max time the query is allowed to run after planning. References • faq/Sql_faq.md • sql-reference/System_variable.md • faq/loading/Loading_faq.md • faq/fe_mem_faq.md • administration/management/FE_configuration.md References • faq/Sql_faq.md • sql-reference/System_variable.md • faq/loading/Loading_faq.md • faq/fe_mem_faq.md • administration/management/FE_configuration.md • faq/Others.md References • knowledge/trouble_shooting/fe_memory_problems.md • faq/Others.md • faq/fe_mem_faq.md • administration/management/FE_configuration.md • faq/operation_maintenance_faq.md • administration/Meta_recovery.md • administration/management/monitoring/Monitor_and_Alert.md
h
@Rocky Does changing this require restarting starrocks FE?
Copy code
new_planner_optimize_timeout
r
No, changing 
new_planner_optimize_timeout
 does not require restarting the StarRocks FE. It is a system variable, which means you can modify it dynamically using the 
SET
command. The change takes effect immediately: * For the current session only:
Copy code
sql
    SET new_planner_optimize_timeout = 10000;
* Globally (for all future sessions):
Copy code
sql
    SET GLOBAL new_planner_optimize_timeout = 10000;
Note: If you set it globally, existing connections will not see the change; only new connections established after the command will use the new value. Difference between "System Variables" and "FE Configs" * System Variables (like this one): Set via SQL, no restart needed. * FE Configurations (found in 
fe.conf
): Usually require a restart unless they are explicitly marked as "mutable" in the documentation. References • sql-reference/System_variable.md • faq/loading/Loading_faq.md • administration/management/FE_configuration.md References • sql-reference/System_variable.md • administration/management/FE_configuration.md • faq/loading/Loading_faq.md
h
@Rocky How to persist the change?
Copy code
SET GLOBAL new_planner_optimize_timeout = 10000;
r
Executing 
SET GLOBAL
is sufficient to persist the change. In StarRocks, when you run 
SET GLOBAL new_planner_optimize_timeout = 10000;
, the value is automatically persisted in the Frontend (FE) metadata (stored in the edit logs and image files). * No 
fe.conf
changes needed: You do not need to add this variable to the 
fe.conf
file. * Survives Restarts: Because it is stored in the metadata, the value will remain 
10000
even if you restart the FE nodes or the entire cluster. * Effect on Sessions: * New sessions: All connections created after the command is run will use the new value. * Existing sessions: Connections that are already open will continue to use the old value. To update an active session without reconnecting, run 
SET new_planner_optimize_timeout = 10000;
(without 
GLOBAL
). Summary of Persistence Action | Persistence | Scope 
SET <var> = ...
| No (Lost after session ends) | Current session only 
SET GLOBAL <var> = ...
| Yes (Stored in FE metadata) | All future sessions References • sql-reference/System_variable.md • administration/management/FE_configuration.md • faq/loading/Loading_faq.md References • sql-reference/System_variable.md • administration/management/FE_configuration.md
h
@Rocky Streamload failed. What are the causes?
Copy code
- SQL connectivity to staging works: DELETE statements on port 9030 complete successfully.
- The subsequent stream-load request to the staging FE/load-balancer on port 8030 fails with:
	('Connection aborted.', ConnectionResetError(54, 'Connection reset by peer'))
r
The error 
ConnectionResetError(54, 'Connection reset by peer')
during a Stream Load is a common networking issue that typically occurs because the TCP connection is abruptly closed by the server-side (the Frontend, Backend, or an intermediate Load Balancer). Since your 
DELETE
statements (metadata operations) work on port 9030 but the 
PUT
request (data transfer) fails on port 8030, the most likely causes are related to how Stream Load handles data and redirection. 1. Network Redirection (The 307 Redirect) StarRocks Stream Load follows a two-step process: 1. The client sends a 
PUT
request to the Frontend (FE) on port 8030. 2. The FE responds with an HTTP 307 Temporary Redirect, directing the client to a specific Backend (BE) node on port 8040. 3. The client then sends the actual data to that BE node. The Problem: If your client or Load Balancer is in a different network than the StarRocks nodes, the FE might be redirecting the client to a private IP address of a BE that the client cannot reach. * Check: Verify if the client can reach the BE nodes directly on port 8040. * Fix: Ensure the BE nodes are configured with reachable IPs in their 
priority_networks
setting, or use a reverse proxy (like Nginx) that handles the redirection for the client. 2. Load Balancer Timeouts and Body Size Limits Because Stream Load involves sending large payloads, many standard Load Balancer configurations will drop the connection if: * 
client_max_body_size
(Nginx) / Payload Limit: The data file exceeds the LB's allowed request size. The LB will send a 
TCP RST
(reset), resulting in your error. * Idle Timeout: The LB closes the connection if it doesn't see progress within a few seconds. 3. Missing 
Expect: 100-continue
Header StarRocks expects the 
Expect: 100-continue
header for Stream Load. This allows the FE to tell the client "Yes, I am ready" (or "Redirect to this BE") before the client starts uploading a massive file. * If your client (e.g., a Python script using 
requests
) starts streaming the body immediately without waiting for the 100-continue response, and the FE tries to send a Redirect or an Error, the connection may be reset by the LB or the FE. * Fix: Ensure your code sends the 
Expect: 100-continue
header. 4. BE Configuration (
streaming_load_max_mb
) If the file you are uploading is larger than the BE's limit (default 10GB), the BE may reject it. * Check: BE configuration 
streaming_load_max_mb
. * Note: Usually, this returns a specific HTTP error message, but in some network environments, it can trigger a connection reset. Troubleshooting Steps 1. Try 
curl
first: Use the official recommended command to rule out client library issues:
Copy code
bash
    curl --location-trusted -u <user>:<pass> -H "Expect:100-continue" \
         -T <file_path> http://<fe_host>:8030/api/<db>/<table>/_stream_load
The 
--location-trusted
flag is critical as it allows 
curl
to follow the 307 redirect and pass credentials to the BE. 2. Verify Port 8040: From the machine running the load, try to 
telnet
or 
nc
to one of the BE nodes on port 8040. If this is blocked, Stream Load will fail after the FE redirect. 3. Check LB Logs: Look at your Load Balancer logs (Nginx/ALB/F5) for any "Request Entity Too Large" or "Connection timed out" errors. References • loading/StreamLoad.md • faq/loading/Stream_load_faq.md • sql-reference/sql-statements/loading_unloading/STREAM_LOAD.md • administration/management/FE_configuration.md • administration/management/resource_management/Load_balance.md
h
@Rocky In what logs can I find the related errors? This is the error from the application
Copy code
- SQL connectivity to staging works: DELETE statements on port 9030 complete successfully.
- The subsequent stream-load request to the staging FE/load-balancer on port 8030 fails with:
	('Connection aborted.', ConnectionResetError(54, 'Connection reset by peer'))
r
To diagnose a 
Connection reset by peer
error during a Stream Load, you should examine logs in the following locations, ordered by the most likely source of the reset: 1. Load Balancer / Proxy Logs (If applicable) If you are using a Load Balancer (like Nginx, HAProxy, or an AWS ALB) on port 8030, this is the most common place where the connection is reset. * What to look for: Look for "413 Request Entity Too Large," "504 Gateway Timeout," or explicit TCP reset messages. * Why: LBs often have a 
client_max_body_size
or idle timeout that is shorter than the time required to stream your data. 2. Frontend (FE) Logs The FE handles the initial handshake and redirects the client to a BE. * 
fe.audit.log
: Check if the 
PUT
request even reached the FE. If it’s not here, the connection was dropped by the network/LB before StarRocks saw it. * 
fe.log
/ 
fe.warn.log
: Search for the 
label
of your load or "StreamLoad". If the FE is rejecting the request (e.g., due to authentication or full task queues), it will be logged here. * Location: 
${STARROCKS_HOME}/log/
3. Backend (BE) Logs Since Stream Load redirects the data flow to a BE node (typically on port 8040), the BE is often the "peer" that resets the connection if it encounters an issue mid-stream. * 
<http://be.INFO|be.INFO>
or 
be.WARNING
: Search for the 
TxnId
or the 
Label
of your load. Look for "fail to receive data" or "close connection" messages. * 
be.out
: If the BE process itself encountered a low-level error or a crash (rare), it might be logged here. * Location: 
${STARROCKS_HOME}/log/
4. BE Error Log (Data Errors) If the connection is reset because of a specific data quality issue that caused the BE to abort: * 
error_log
: Check the directory 
${STARROCKS_HOME}/storage/error_log
. These files contain specific rows that failed validation. Summary Checklist for your specific error: 1. Check 
fe.audit.log
: Did the FE receive the request? 2. Check 
<http://be.INFO|be.INFO>
on all BEs: Search for the load label. If you see "connection closed by client," the problem is on your application/network side. If you see a BE-side error, that is the cause. 3. Validate Port 8040: The most common cause for "Connection Reset" after the initial request is that the client cannot reach the Backend IP on port 8040 after being redirected by the FE. References • administration/management/logs.md • loading/loading_introduction/troubleshooting_loading.md • faq/loading/Stream_load_faq.md • loading/StreamLoad.md References • administration/management/logs.md • loading/loading_introduction/troubleshooting_loading.md • faq/loading/Stream_load_faq.md • loading/StreamLoad.md References • administration/management/logs.md • loading/loading_introduction/troubleshooting_loading.md
h
@Rocky Streamload failed. It went through an AWS ELB
Copy code
- SQL connectivity to staging works: DELETE statements on port 9030 complete successfully.
- The subsequent stream-load request to the staging FE/load-balancer on port 8030 fails with:
	('Connection aborted.', ConnectionResetError(54, 'Connection reset by peer'))
r
When using an AWS ELB (especially an Application Load Balancer/ALB) for StarRocks Stream Load, a 
ConnectionResetError
on port 8030 usually occurs because of how the ELB handles the HTTP 307 Redirect or the large data payload. Here are the most common causes and how to fix them: 1. The "Unreachable BE" Problem (307 Redirect) StarRocks Stream Load is a two-step process: 1. Handshake: Your application sends a 
PUT
request to the ELB (port 8030). 2. Redirect: The FE responds with an HTTP 307 Temporary Redirect, providing the private IP and port (8040) of a Backend (BE) node. 3. Data Transfer: Your application then tries to connect directly to that BE IP. Why it fails: If your application is running outside the VPC (e.g., on your local machine or a different network), it cannot reach the BE's private IP. When the client attempts this connection and fails, or if the ELB resets the connection because it doesn't support the redirected flow, you get a 
Connection Reset
. * Solution: Use a Network Load Balancer (NLB) instead of an ALB if possible. NLBs are more transparent for StarRocks traffic. * Alternative: If using an ALB, ensure your client is within the same VPC as the StarRocks cluster, or use a proxy that handles the redirection (like Nginx) to keep the traffic within the ELB. 2. Missing 
Expect: 100-continue
Header StarRocks requires the 
Expect: 100-continue
header. * The Issue: Without this header, your application may start streaming the data immediately. If the FE/LB tries to send the 307 Redirect while the client is still pushing data, the TCP window can become desynchronized, leading the AWS ELB to send a 
TCP RST
(Reset). * Solution: In Python (Requests), this is not always enabled by default for all versions. In 
curl
, it is automatic. Ensure your application explicitly waits for the 
100 Continue
response before sending the body. 3. AWS ALB Idle Timeout The default AWS ELB idle timeout is 60 seconds. * The Issue: If your Stream Load takes longer than 60 seconds to stream or if there is a lull in data packets (common with large files or complex transformations), the ELB will abruptly close the connection. * Solution: Increase the Idle Timeout in your AWS ELB attributes to a higher value (e.g., 3600 seconds). 4. BE Port 8040 is Blocked The ELB might only be configured to listen on port 8030. * The Issue: After the 307 redirect, the client needs to talk to the BE on port 8040. If your Security Groups only allow 8030 (FE) and 9030 (MySQL), the connection to the BE will be blocked, causing a timeout or reset. * Solution: Ensure the Security Group for your BE nodes allows inbound traffic on port 8040 from your application/ELB. Troubleshooting Steps 1. Check Redirection: Run the load using 
curl -v --location-trusted ...
. Look at the 
Location:
header in the response. If it contains a private IP (e.g., 
10.x.x.x
), your client must be able to route to that IP. 2. Verify direct BE access: From the application server, try 
telnet <BE_IP> 8040
. If it fails, the redirection will never work. 3. Bypass the LB: If possible, try to Stream Load directly to a BE's IP on port 8040 to see if the error persists. If it works directly but fails via ELB, the ELB configuration (timeout or header handling) is the culprit. References • loading/StreamLoad.md • faq/loading/Stream_load_faq.md • administration/management/FE_configuration.md • administration/management/resource_management/Load_balance.md References • quick_start/helm.md
h
@Rocky Streamload failed. It went through an AWS NLB. It worked before
Copy code
- The subsequent stream-load request to the staging FE/load-balancer on port 8030 fails with:
	('Connection aborted.', ConnectionResetError(54, 'Connection reset by peer'))
r
If your Stream Load was working and is now failing with a 
ConnectionResetError
through an AWS NLB, the issue is usually related to how the NLB manages TCP sessions, target health, or the specific network path to the Backends (BEs). Since it "worked before," look for changes in your environment related to the following four areas: 1. The 350-Second NLB Idle Timeout AWS Network Load Balancers have a fixed idle timeout of 350 seconds (approx. 5.8 minutes) that cannot be changed. * The Problem: If your data file has grown or your network speed has slowed, and the connection remains "idle" (no TCP packets sent) for more than 350 seconds, the NLB will drop the connection and send a 
TCP RST
(Reset). * Why it worked before: Your previous loads likely finished within this window. * The Fix: * Enable TCP Keepalives on your client-side (e.g., in your Python 
requests
or 
urllib3
configuration). * If the file is large, split it into smaller chunks or increase the 
streaming_load_max_mb
and speed up the transfer. 2. Unhealthy Targets or Cluster Scaling If you recently added, removed, or restarted nodes, the NLB may be resetting connections. * The Problem: When an NLB target (FE) is marked unhealthy, the NLB immediately sends a 
TCP RST
for any packets received on existing client connections to that target. * Check: Go to the AWS Console -> Target Groups -> Your FE Target Group. Ensure all FEs are 
Healthy
. Check the 
TCP_ELB_Reset_Count
metric in CloudWatch; a spike here confirms the NLB is the source of the reset. * Cross-Zone Balancing: If you have FEs in multiple Availability Zones but "Cross-Zone Load Balancing" is disabled (the default for NLB), and a node in one zone goes down, clients hitting that zone's IP will get resets. 3. Port 8040 & Security Group Changes Stream Load is a two-step process. The NLB handles the request to port 8030, but the FE then redirects the client to a BE node. 1. Request: Client -> NLB (8030) -> FE (8030). 2. Redirect: FE returns a 
307 Temporary Redirect
to a BE IP on port 8040. 3. Data: Client -> BE (8040). * The Problem: If a Security Group rule was recently tightened and port 8040 is no longer open to your client, the connection will fail. * Why it resets: Sometimes the "Reset" is actually the client or an intermediate proxy giving up when the BE is unreachable. * The Fix: Ensure port 8040 is open from the client to all BE private/public IPs. 4. Client IP Preservation & Asymmetric Routing By default, NLBs with "Instance" targets preserve the Client IP. * The Problem: If you recently changed VPC routing, NACLs, or added a NAT Gateway, the return traffic from the FE/BE might be trying to take a different path than the inbound traffic (Asymmetric Routing). AWS will drop these packets and reset the connection. * The Fix: If your client is in a different VPC or outside AWS, try disabling "Client IP Preservation" on the NLB Target Group attributes (this changes the source IP to the NLB's private IP, which often fixes routing loops). Recommended Troubleshooting Steps 1. Bypass the NLB: Try to run the 
curl
command directly against an FE's private/public IP. If it works, the issue is strictly the NLB configuration. 2. Use `curl -v`: Look for the 
Location
header in the response. Check if that IP:Port (usually 8040) is reachable from your application server. 3. Check FE Logs: Look at 
fe.warn.log
or 
fe.audit.log
. If you see the request there, the FE received it and the reset happened after the handshake (likely during the redirect or data transfer). References • loading/StreamLoad.md • faq/loading/Stream_load_faq.md • administration/management/FE_configuration.md
h
@Rocky Steamload failed with this error. It worked before with smaller data. The application sent the streamload to an AWS NLB and got the IP of the BE from the FE, then sent the streamlaod to the BE
Copy code
- The subsequent stream-load request to the staging FE/load-balancer on port 8030 fails with:
	('Connection aborted.', ConnectionResetError(54, 'Connection reset by peer'))
r
Since your Stream Load worked with smaller data and you are using an AWS NLB, the 
ConnectionResetError
(54) with larger data is almost certainly caused by one of three specific "threshold" issues related to time, memory, or network routing. 1. The 350-Second NLB Idle Timeout The most common cause for "Connection Reset" with an AWS NLB is its fixed 350-second (5.8 minute) idle timeout. * The Issue: If your data transfer takes longer than 350 seconds, or if there is a period of "silence" while the BE is processing a large batch (e.g., during a memory flush), the NLB will silently drop the connection. When your application or the StarRocks BE tries to send the next packet, the NLB responds with a 
TCP RST
(Reset). * The Fix: * Enable TCP Keepalives: Ensure your client library (e.g., Python 
requests
or Java 
HttpClient
) has TCP keepalives enabled. This sends small packets to keep the NLB session active. * Split the Data: Instead of one massive load, split the data into smaller chunks (e.g., 1-2 GB each) that complete within 5 minutes. * Check NLB Metrics: In AWS CloudWatch, check the 
TCP_ELB_Reset_Count
metric. A spike here during your load confirms the NLB is the culprit. 2. StarRocks Internal Timeouts With larger data, you may be hitting StarRocks' own safety limits. * 
stream_load_default_timeout_second
(FE): The default is 600 seconds. If the load takes longer than this, the FE will abort the transaction, and the BE will close the connection. * Solution: Pass a higher 
timeout
header in your request: 
-H "timeout:3600"
. * 
streaming_load_max_mb
(BE): If your data exceeds this limit (default 10GB or 100GB depending on version), the BE may abort. * Solution: Verify this in 
be.conf
or check the 
<http://be.INFO|be.INFO>
logs for "body exceed max size". 3. Asymmetric Routing and MTU (NLB Specific) Larger data transfers involve significantly more packets and higher throughput, which can expose network configuration flaws that smaller loads don't trigger. * Client IP Preservation: If your NLB has "Client IP Preservation" enabled and your BEs are in a private subnet, the return traffic from the BE might attempt to bypass the NLB and go directly to the client. This "Asymmetric Routing" causes the client's OS to reset the connection because it's receiving packets from an IP it didn't initiate a connection with. * Solution: In the NLB Target Group settings, try disabling Client IP Preservation. * MTU Mismatch: Large data packets (especially if using Jumbo Frames 9001 MTU inside AWS) that cross into a network with a standard 1500 MTU can be dropped if "Fragmentation" isn't handled correctly, leading to a reset. 4. BE Memory Pressure Large loads require significant memory for buffering and sorting (controlled by 
write_buffer_size
in 
be.conf
, default 100MB). * The Issue: If the load is so large that it pushes the BE over its 
load_process_max_memory_limit_percent
(default 30% of BE memory), the BE will kill the process to protect the system, resulting in an immediate connection reset. * Check: Search the 
be.WARNING
logs for "memory limit exceeded" or "cancel load job". Recommendation: How to identify the culprit 1. Check FE Logs (
fe.audit.log
and 
fe.warn.log
): If the load is logged with a status like 
CANCELLED
and a message "timeout," increase the 
timeout
header. 2. Check BE Logs (
<http://be.INFO|be.INFO>
): Search for the 
Label
of your load. If you see "connection closed by peer," the NLB or Client timed out. If you see an internal error followed by "close connection," the BE is the source. 3. Bypass the NLB: If possible, try the same large load directly against a BE IP on port 8040. If it works, the issue is 100% the AWS NLB configuration (Timeout or Client IP Preservation). References • loading/StreamLoad.md
h
@Rocky Steamload failed with this error. It worked before with smaller data. The application sent the streamload to an AWS NLB and got the IP of the BE from the FE, then sent the streamload directly to the BE IP address
Copy code
- The subsequent stream-load request to the staging FE/load-balancer on port 8030 fails with:
	('Connection aborted.', ConnectionResetError(54, 'Connection reset by peer'))
r
If your Stream Load is failing with a 
ConnectionResetError
(Error 54) on port 8030 (the FE/NLB) specifically during "subsequent" requests after successfully processing smaller data, the issue is likely rooted in how the AWS Network Load Balancer (NLB) manages TCP state and connection persistence. Because it worked for smaller data but fails now, and occurs on "subsequent" attempts, here are the most likely causes and solutions: 1. NLB 350-Second Idle Timeout (Most Likely) AWS NLBs have a hard-coded 350-second (5.8 minute) idle timeout. * The Issue: When you perform a Stream Load, your application establishes a connection to the FE (port 8030) to get the redirect to a BE. Even if you then send the data "directly" to the BE IP, your application's HTTP client library often keeps the original connection to the FE/NLB open in a connection pool for re-use. * The Failure: If your large data transfer to the BE takes longer than 350 seconds, the NLB will silently drop the "idle" connection to the FE. When your application tries to start the next Stream Load by re-using that pooled connection, the NLB sees traffic for a session it no longer tracks and sends a 
TCP RST
(Reset), resulting in the 
ConnectionResetError
. * The Fix: * Disable Connection Pooling: Configure your client (e.g., Python 
requests.Session()
) to close the connection after each load, or set a 
max_idle_time
for the pool significantly lower than 300 seconds. * Enable TCP Keepalives: Ensure your OS and client library are sending TCP keepalive packets to prevent the NLB from timing out the session. 2. Missing 
Expect: 100-continue
Header StarRocks expects the 
Expect: 100-continue
header for Stream Loads. * The Issue: For larger data, if your client sends the HTTP headers and starts pushing the request body before the FE has a chance to process the handshake/redirect, the NLB can become overwhelmed or desynchronized. If the FE closes the socket to send the 307 Redirect while the client is still pushing data, the NLB will reset the connection. * The Fix: Ensure your request explicitly includes the header 
-H "Expect: 100-continue"
. This forces the client to wait for a "Go ahead" from the FE before it sends any part of the body. 3. NLB "Diamond Routing" (Flow Collision) If you have Cross-Zone Load Balancing and Client IP Preservation both enabled on your NLB, you can hit a known AWS issue called "Flow Collision." * The Issue: High-throughput loads from a single client can cause the NLB to map different connections to the same internal backend using the same 5-tuple (source IP/port, dest IP/port). This causes the NLB to reset the "colliding" connection. * The Fix: 1. Go to your NLB Target Group attributes. 2. Disable "Client IP Preservation" OR Disable "Cross-Zone Load Balancing". _ _Note: Disabling Client IP Preservation is usually the safer fix as it changes the source IP to the NLB's private IP, avoiding routing collisions.* 4. StarRocks Transaction Timeout While the error appears on port 8030, it could be triggered by an internal FE timeout. * The Issue: If the large data load takes longer than the FE's 
stream_load_default_timeout_second
(default 600s), the FE will abort the transaction. If your client is still trying to communicate with the cluster via the LB during this abort, the connection can be reset. * The Fix: Increase the timeout in your request header: 
-H "timeout:3600"
. Recommended Troubleshooting 1. Verify via `curl`: Run the load using 
curl -v --location-trusted ...
. If 
curl
works while your application fails, your application's connection pooling is the culprit. 2. Check NLB Metrics: In CloudWatch, look for 
TCP_ELB_Reset_Count
. If this spikes exactly when your load fails, the NLB is explicitly closing the connection.
3. Check FE Logs: Search 
fe.audit.log
for the 
Label
of the failed load. If it doesn't even appear in the audit log, the connection was reset by the NLB before it ever reached the StarRocks FE. References • faq/operation_maintenance_faq.md • quick_start/helm.md • faq/loading/Stream_load_faq.md • loading/StreamLoad.md • administration/management/monitoring/metrics.md • sql-reference/sql-statements/loading_unloading/STREAM_LOAD.md • loading/Stream_Load_transaction_interface.md
h
@Rocky Steamload failed with this error that came from the BE. It worked before with smaller data. The application sent the streamload to an AWS NLB and got the IP of the BE from the FE, then sent the streamload directly to the BE IP address
Copy code
('Connection aborted.', ConnectionResetError(54, 'Connection reset by peer'))
r
If your Stream Load fails with a ConnectionResetError (54) during a large data transfer—especially when the error message mentions a "subsequent" request to *port 8030*—the issue is likely a side effect of how your application handles connection pooling in combination with the AWS NLB idle timeout. Even though you are sending data "directly" to the BE IP, the root cause often lies in the original connection established through the NLB to the FE. 1. NLB Connection Pool Timeout (Most Likely) When you perform a Stream Load, your application first connects to the NLB on port 8030 to get a redirect. Most modern HTTP clients (like Python's 
requests
or Java's 
HttpClient
) keep this connection open in a pool to reuse it for the next request. * The Problem: While your application is busy sending large amounts of data "directly" to the BE IP, the initial connection to the NLB/FE (port 8030) sits idle. * The Failure: AWS NLBs have a fixed 350-second idle timeout. If your direct data transfer to the BE takes longer than ~5.8 minutes, the NLB silently drops the idle connection to the FE. * The Result: When your application finishes the first load and attempts a "subsequent" load, it tries to reuse the now-dead connection in its pool. The NLB responds with a 
TCP RST
(Reset), leading to the 
ConnectionResetError
. * The Fix: * Disable Connection Pooling: In your application, set the HTTP headers to 
Connection: close
for the request to the FE, or configure your client library to not persist connections. * Shorten Pool TTL: Set your connection pool's "Max Idle Time" to something lower than 300 seconds. 2. BE Memory Pressure (BE-Side Reset) If you are certain the reset is happening during the data transfer to the BE (port 8040), the BE may be abruptly closing the connection because it ran out of memory or hit a safety limit. * 
load_process_max_memory_limit_percent
: By default, StarRocks limits the memory used by all loading tasks to 30% of the BE's total memory. If a very large load pushes the BE over this limit, the BE will kill the loading thread to protect the system, which causes an immediate TCP Reset. * Check: Look at the 
be.WARNING
or 
<http://be.INFO|be.INFO>
logs for the string 
memory limit exceeded
or 
cancel load job
. * 
streaming_load_max_mb
: The default is 10 GB (pre-v3.0) or 100 GB (v3.0+). If your data exceeds this, the BE will stop reading and close the socket. * Check: Search BE logs for 
body exceed max size
. 3. MTU and Segment Size Issues Large data transfers result in many full-sized TCP segments. * The Issue: If there is an MTU mismatch between your application server and the BE (e.g., your app server is using a standard 1500 MTU but the BE/VPC is configured for Jumbo Frames 9001, or vice versa), large packets may be dropped or rejected by network interfaces or intermediate security groups, resulting in a reset. * The Fix: Ensure the MTU is consistent across your network path (usually 1500 is the safest for cross-VPC or external traffic). 4. StarRocks Transaction Timeout * The Issue: If the large load takes longer than the FE's 
stream_load_default_timeout_second
(default 600 seconds), the FE will abort the transaction and notify the BE. The BE will then close the connection. * The Fix: Increase the timeout in your request header: 
-H "timeout:3600"
(setting it to 1 hour). Summary of Actions 1. Check FE Audit Logs: Look for the "Label" of the failed load. If the state is 
CANCELLED
with a 
type: TIMEOUT
, increase your 
timeout
header. 2. Check BE Info Logs: Search for 
Connection reset by peer
or 
memory limit exceeded
. 3. App Configuration: If using Python 
requests
, try using a fresh 
Session()
or explicitly closing the connection after the redirect step to ensure the NLB timeout doesn't poison your pool. References
• faq/loading/Stream_load_faq.md • loading/StreamLoad.md • administration/management/BE_configuration.md • loading/loading_introduction/loading_considerations.md • faq/loading/Loading_faq.md References • faq/operation_maintenance_faq.md • loading/StreamLoad.md • faq/loading/Stream_load_faq.md • administration/management/BE_configuration.md • administration/management/FE_configuration.md • loading/loading_introduction/loading_considerations.md • faq/loading/Loading_faq.md • administration/management/FE_configuration.md References • administration/management/BE_configuration.md • loading/StreamLoad.md • faq/loading/Stream_load_faq.md • loading/loading_introduction/loading_considerations.md • faq/loading/Loading_faq.md • administration/management/FE_configuration.md
h
@Rocky Steamload failed with the error and the follow-up test showed this
Copy code
- The subsequent stream-load request to the staging FE/load-balancer on port 8030 fails with:
	('Connection aborted.', ConnectionResetError(54, 'Connection reset by peer'))
	
This provides stronger evidence that the issue is intermittent routing/connectivity before the FE redirects to a BE:

- The same 10-row chunk can fail with `ConnectionResetError(54)` and then succeed on a retry.
- Therefore it is not caused by payload size or by a malformed row in that chunk.
- Every failed attempt resets before the FE returns a redirect, so no BE IP is available for those failed attempts.
r
The evidence from your tests confirms that the connection is being reset at the network/load-balancer layer (port 8030) before it even reaches the StarRocks Frontend (FE) logic. Since it occurs intermittently with small data (10 rows) and "subsequent" to larger loads, your issue is caused by two specific AWS NLB behaviors: Connection Pool Poisoning and Flow Collisions. 1. Connection Pool Poisoning (The "Subsequent" Failure) This explains why a load fails after a previously successful one. * The Mechanism: When your application starts a Stream Load, it connects to the NLB (port 8030). Even if you then send data directly to the BE, your HTTP client library (e.g., Python 
requests
or Java 
HttpClient
) usually keeps that initial "FE connection" open in a connection pool for future use. * The Trigger: AWS NLBs have a hard-coded 350-second idle timeout. If your direct-to-BE data transfer takes longer than 5.8 minutes, the NLB silently drops the idle connection to the FE. * The Error: When your app attempts a "subsequent" load, it pulls that "dead" connection from its pool. The NLB, having discarded the state, responds with a 
TCP RST
, causing the 
ConnectionResetError(54)
. * The Fix: * Disable connection pooling in your application for the redirect request. * Or, set your application's Connection Max Idle Time to 60 seconds (well below the 350s limit). 2. Flow Collisions / Diamond Routing (The "Intermittent" Failure) This explains why a 10-row chunk might fail once but succeed on a retry. * The Mechanism: This happens when an NLB has both Client IP Preservation and Cross-Zone Load Balancing enabled. * The Trigger: If multiple connections from the same client IP/port attempt to reach the FE, the NLB can occasionally map a new connection to an internal "flow" that is still being cleaned up or exists in another AZ. This results in a "Flow Collision," where the NLB sends an immediate 
TCP RST
. * The Fix: 1. Go to the AWS Target Group settings for your FE (port 8030). 2. Disable "Client IP Preservation". This causes the FE to see the NLB's private IP instead of the client's, which resolves the routing collision. 3. Alternatively, Disable "Cross-Zone Load Balancing", though disabling Client IP Preservation is the standard fix for intermittent resets. 3. StarRocks FE Backlog (Less Likely but Possible) If the NLB is healthy, the FE itself might be dropping the connection if it cannot handle the initial handshake fast enough. * Check: In your 
fe.warn.log
, look for "connect processor exception" or "IOException: Connection reset by peer." * Fix: If you see these, you may need to increase the OS-level backlog on the FE host:
Copy code
bash
    # Increase the maximum number of pending connections
    sysctl -w net.core.somaxconn=1024
Summary of Recommended Actions 1. NLB Level: Disable Client IP Preservation in the FE Target Group. 2. Application Level: Ensure you are not reusing connections that have been idle for more than 300 seconds. If using Python, use a new 
requests.Session()
or set 
headers={'Connection': 'close'}
for the initial request to port 8030. 3. CloudWatch Metrics: Monitor the 
TCP_ELB_Reset_Count
metric. If it spikes during these failures, it confirms the NLB is the source of the 
RST
packets. References • faq/operation_maintenance_faq.md • loading/StreamLoad.md • faq/loading/Stream_load_faq.md • administration/management/FE_configuration.md References • faq/operation_maintenance_faq.md • faq/loading/Stream_load_faq.md
h
@Rocky Streamload failed at this error. The application sent the streamload to a FE IP address. FE never sent back the redirect to BE
Copy code
('Connection aborted.', ConnectionResetError(54, 'Connection reset by peer'))
r
Since the error is occurring at the FE IP (port 8030) before a redirect is even issued, and given that the issue is intermittent and specifically affects larger data or "subsequent" loads, the cause is likely a protocol mismatch or a resource limit on the FE. 1. Missing 
Expect: 100-continue
(Most Common) This is the most frequent cause of 
ConnectionResetError
when sending Stream Loads directly to an FE. * The Issue: StarRocks FEs do not want to receive the data body; they only want the headers so they can decide which BE should handle the load. If your application sends the HTTP request headers and immediately starts streaming the data body, the FE will attempt to send a 
307 Temporary Redirect
and close the connection. * The Reset: If the FE closes the socket while your application is still pushing the data body, the FE's network stack sends a 
TCP RST
(Reset). Your application receives this reset before it can process the 307 Redirect response, making it appear as if the FE "never sent the redirect." * The Fix: Ensure your HTTP client is configured to use the 
Expect: 100-continue
header. This forces the client to send the headers, wait for a 
100 Continue
from the FE, and then send the data (or follow the redirect). 2. FE HTTP Thread Pool Exhaustion The FE uses a worker thread pool to handle incoming HTTP requests on port 8030. * The Issue: If you have many concurrent loads or high-latency network connections, the FE might exhaust its 
http_worker_threads_num
. When the pool is full, the FE may drop new connections or take so long to respond that the client/LB resets. * The Fix: * Check 
fe.conf
for 
http_worker_threads_num
. The default is typically 
0
(which scales to 2x CPU cores). You can try increasing this to 
1024
or higher if you have high concurrency. * Check for 
qe_max_connection
(default 4096), although this primarily limits MySQL/Query connections. 3. OS-Level TCP Backlog (somaxconn) If the FE is receiving bursts of requests, the operating system's listen backlog might be overflowing. * The Issue: When the number of pending TCP connections exceeds the kernel limit, the OS will reject or reset new connections. * Verification: Run the following on the FE host:
Copy code
bash
    netstat -s | grep TCPBacklogDrop
If the count is increasing, the backlog is too small. * The Fix: Increase the 
somaxconn
limit on the FE host:
Copy code
bash
    sysctl -w net.core.somaxconn=2048
4. FE Transaction Limits StarRocks has internal limits on the number of concurrent transactions. * 
max_running_txn_num_per_db
: Default is 1000. If your application is starting thousands of transactions simultaneously, the FE will reject new ones. * 
stream_load_max_txn_num_per_be
 (v3.3+): If set, this limits how many loads a single "BE" (or client IP seen by the FE) can have in progress. * Check: Look in 
fe.warn.log
for messages like 
streamload txn num per be exceeds limit
or 
current running txns exceed limit
. 5. FE Memory/GC Pressure If the FE is undergoing a Full GC or is low on memory, it may become unresponsive and drop connections. * Verification: Search 
fe.log
for 
Full GC
or 
java.lang.OutOfMemoryError
. * The Fix: Ensure the FE has sufficient JVM heap (at least 8GB-16GB for production) and check 
fe.conf
for 
-Xmx
settings. Summary Checklist 1. Headers: Verify your app sends 
Expect: 100-continue
. 2. FE Logs: Search 
fe.warn.log
for 
connect processor exception
or 
IOException: Connection reset by peer
. 3. Audit Logs: Check 
fe.audit.log
. If the failed request doesn't appear there at all, the reset happened at the TCP/Handshake layer (OS Backlog or Network). If it appears with a status, the FE logic rejected it. References • faq/operation_maintenance_faq.md
h
@Rocky Streamload failed at this error. The application sent the streamloads to a FE IP address. FE never sent back any redirect to BE. All of the streamloads failed
Copy code
('Connection aborted.', ConnectionResetError(54, 'Connection reset by peer'))
r
If all Stream Loads are failing with a 
ConnectionResetError
at the FE IP address before a redirect is issued, the problem is almost certainly a protocol conflict between your application's HTTP client and the StarRocks FE's HTTP server. The most likely causes are: 1. Missing 
Expect: 100-continue
Header (Most Common) When you send a Stream Load to an FE, the FE reads only the headers to decide which BE should handle the data. It then immediately sends a 
307 Temporary Redirect
response. * The Conflict: If your application (especially if using Python 
requests
) starts sending the data body immediately after the headers without waiting for the FE's permission, the FE will send the redirect and close the socket while your application is still pushing data. * The Result: When a server closes a socket while the client is still sending data, the network stack triggers a TCP RST (Reset). Your application receives 
ConnectionResetError(54)
and fails before it can even see the 
307 Redirect
message. * The Fix: You must configure your HTTP client to use the 
Expect: 100-continue
header. This forces the client to send headers first and wait for the FE to say "Continue" (or "Redirect") before it starts uploading the payload. _ _Python Example:* 
requests.put(url, data=data, headers={'Expect': '100-continue'})
2. FE Not in a "Leader" or "Providing Service" State If the FE you are hitting is not the Leader or is having metadata issues, it may abruptly close the connection. * Check FE Status: Run 
SHOW FRONTENDS;
in your MySQL client. * Verify that 
IsMaster
(Leader) is 
true
for at least one node. * Verify 
Alive
is 
true
for the node you are targeting. * Check IP Mismatch: If the FE host has multiple network interfaces, verify the 
priority_networks
in 
fe.conf
. If the FE is bound to a different IP than what the client is hitting, it may reset connections. 3. FE HTTP Thread Pool Exhaustion If the FE’s HTTP server is overwhelmed, it will reject new connections immediately. * The Fix: Check 
fe.conf
for the following parameters and consider increasing them: * `http_worker_threads_num`: Default is 
0
(scales with CPU). Try setting it to 
1024
. * Check FE Logs: Look at 
fe.warn.log
for 
java.io.IOException: Connection reset by peer
or 
Too many open files
. 4. OS-Level TCP Backlog (
somaxconn
) If you are sending many requests at once, the Linux kernel's queue for "pending" connections might be full, causing the OS to reset the connection before the StarRocks process even sees it. * The Fix: Increase the system-wide limit on both the FE and the client machine:
Copy code
bash
    # Check current limit
    cat /proc/sys/net/core/somaxconn
    # Increase to 2048
    sysctl -w net.core.somaxconn=2048
5. Incorrect URL Syntax Ensure you are hitting the specific Stream Load API endpoint. If you hit a generic FE port with the wrong path, the FE may close the connection. * Correct Syntax: 
http://<fe_ip>:8030/api/<db>/<table>/_stream_load
Diagnostic Steps 1. Test with `curl`: Run a test from the application server using 
curl
. Curl handles the 
Expect: 100-continue
and redirect logic automatically:
Copy code
bash
    curl --location-trusted -u user:pass -H "Expect:100-continue" \
         -T test_file.csv http://<FE_IP>:8030/api/<db>/<table>/_stream_load
If 
curl
works but your app fails, the issue is your application's HTTP header/pooling configuration. 2. Check FE Audit Log: Look at 
fe/log/fe.audit.log
. If you do not see the failed requests listed there, the connection is being reset by the OS or Network before the StarRocks application can log it. References • sql-reference/sql-statements/loading_unloading/STREAM_LOAD.md • faq/operation_maintenance_faq.md
2 Views
Open in Slack
PreviousNext
Powered by