http://starrocks.io logo
Join Slack
Powered by
This message was deleted.
# questions-and-troubleshooting
s
This message was deleted.
r
When connecting StarRocks to a "huge" database using the AWS Glue Data Catalog, performance and stability challenges typically arise from metadata volume (many partitions/files) and I/O throughput (S3 scanning). Below are the key FE and CN/BE configurations to tune for large-scale AWS Glue environments. 1. FE (Frontend) Configurations The FE handles metadata caching and query planning. For large Glue catalogs, the priority is preventing OOM during planning and speeding up partition pruning. JVM Memory (fe.conf) * 
JAVA_OPTS
: Increase the heap size (e.g., 
-Xmx32g
or higher). Large Glue databases with hundreds of thousands of partitions require significant memory to store and process the metadata during planning. Metadata Caching & Background Refresh These are dynamic parameters set via 
ADMIN SET FRONTEND CONFIG
. * 
enable_background_refresh_connector_metadata
: Set to 
true
. This allows StarRocks to proactively pull Glue metadata in the background so queries don't hang waiting for Glue API calls. * 
background_refresh_metadata_interval_millis
: Controls how often metadata is refreshed (default 10 mins). For very large databases, you might increase this to reduce Glue API costs/throttling. * 
iceberg_metadata_memory_cache_capacity
: (If using Iceberg) Increase this if you have many large metadata files. Catalog Properties (at CREATE CATALOG) When creating the catalog, use these properties to optimize metadata handling: * 
enable_metastore_cache
: 
true
(default). * 
enable_cache_list_names
: 
true
. Caches the list of partition names to speed up pruning. * 
metastore_cache_refresh_interval_sec
: Determines how long metadata stays fresh in the local cache. Query Planning (Session Variables) * 
plan_mode
: For very large Iceberg or Hive tables, set this to 
distributed
. This allows BE/CN nodes to help the FE parse metadata files (like Iceberg manifests), preventing the FE from becoming a bottleneck. ────────── 2. CN / BE (Compute Node / Backend) Configurations The CN/BE nodes handle the actual data scanning from S3. Scan Parallelism (be.conf) * 
pipeline_connector_scan_thread_num_per_cpu
: Controls the number of threads per CPU core assigned to scan external data. For high-throughput S3 reads, you can increase this (e.g., to 
16
or 
32
). * 
max_hdfs_scanner_num
: Limits the maximum number of concurrent remote scanners. Increase this if your hardware can handle more parallel I/O. I/O & S3 Optimization * 
connector_io_tasks_per_scan_operator
: Increases the number of concurrent I/O tasks for each scan operator. * AWS S3 Timeouts: If you encounter timeouts during heavy scans, adjust these in 
be.conf
or as catalog properties: * 
aws.s3.connect_timeout_ms
* 
aws.s3.socket_timeout_ms
────────── 3. Summary Checklist for "Huge" Scale Component | Parameter | Recommended Action FE | 
JAVA_OPTS
(
-Xmx
) | Increase to 32GB-64GB+ FE | 
enable_background_refresh_connector_metadata
| Set to 
true
FE | 
max_partition_number_per_table
| Ensure this is higher than your largest table's partition count (Default: 100,000) CN/BE | 
pipeline_connector_scan_thread_num_per_cpu
| Increase for higher S3 throughput Catalog | 
aws.glue.region
| Ensure it matches your Glue/S3 region to avoid cross-region latency Note: If your database is "huge" due to the number of small files, consider using Data Cache (StarRocks' local disk cache) on CN nodes to avoid repeated S3 fetches. Configure 
storage_root_path
with 
medium:SSD
in 
be.conf
. References • using_starrocks/caching/query_cache.md • data_source/catalog/deltalake_catalog.md • data_source/catalog/hive_catalog.md
For an FE node on a machine with 32 GB of total RAM, the best practice is to allocate approximately 75%–80% of that memory to the JVM heap while ensuring that 
-Xms
and 
-Xmx
are equal to prevent performance hits from heap resizing. Based on StarRocks' official recommendations and common production tuning, here is the suggested configuration: Suggested 
JAVA_OPTS
Copy code
bash
JAVA_OPTS="-Dlog4j2.formatMsgNoLookups=true \
-Xms24576m -Xmx24576m \
-XX:+UseG1GC \
-XX:MaxDirectMemorySize=4096m \
-XX:MaxMetaspaceSize=512m \
-XX:+ExitOnOutOfMemoryError \
-Xlog:gc*:${LOG_DIR}/fe.gc.log.$DATE:time"
Key Improvements Explained: 1. Set 
-Xms
equal to 
-Xmx
(24 GB): Your original config had a large gap (8GB to 22GB). StarRocks recommends keeping these consistent (at ~24GB for a 32GB machine) to avoid the overhead of the JVM frequently requesting memory from the OS during metadata-heavy operations (like Glue catalog syncing). 2. Heap Size (24 GB): 24 GB represents roughly 75% of your total 32 GB. While the 80% rule (25.6 GB) is common, 24 GB is "safer" because it leaves ~8 GB for: * Direct Memory: 4 GB (used for network I/O and serialized metadata). * Metaspace: 512 MB. * OS/Thread Stacks: The remaining ~3.5 GB is necessary for the operating system and the hundreds of threads the FE runs. 3. G1GC: Retained. This is the standard for StarRocks FE as it handles large heaps and metadata objects with lower pause times. 4. ExitOnOutOfMemoryError: Retained. This is critical for high availability; if an FE hits an OOM, it’s better for the process to die and be restarted (or for the Leader to failover) than to hang in a zombie state. Important Complementary Setting In addition to 
JAVA_OPTS
, ensure you set the following in your 
fe.conf
 (outside of the JAVA_OPTS string) to prevent the OS from over-allocating memory to glibc's memory arenas, which can lead to OOM kills by the Linux kernel:
Copy code
bash
# Add this to fe.conf
export MALLOC_ARENA_MAX=1
When to go higher? If your AWS Glue catalog is exceptionally large (millions of tablets/partitions), and you notice "GC overhead limit exceeded" errors in 
fe.warn.log
, you may need to upgrade the physical machine to 64GB+ RAM and scale the heap accordingly. • faq/Others.md • administration/management/FE_configuration.md References • administration/management/BE_configuration.md
To connect StarRocks to a "huge" AWS Glue database (e.g., millions of partitions or files), you must optimize for metadata memory management on the FE and I/O throughput on the CN/BE. Below is the recommended full configuration for a large-scale deployment. 1. Hardware Specifications For a "huge" database, FE memory is the most critical factor to prevent OOM during query planning. Node Type | Recommended CPU | Recommended RAM | Storage FE | 16 Cores | 64GB - 128GB+ | 200GB+ SSD (for metadata logs/cache) CN / BE | 16 - 32 Cores | 64GB - 128GB | 1TB+ NVMe (for Data Cache) ────────── 2. FE Configuration (
fe.conf
) Modify these parameters to handle massive metadata volumes from Glue/S3. Parameter | Recommended Value | Description 
JAVA_OPTS
 | 
-Xmx64g -Xms64g
| Match heap size to your scale. 1M+ partitions usually need 64GB+. 
max_partition_number_per_table
 | 
500000
| Default is 100k. Increase if tables have more partitions. 
enable_background_refresh_connector_metadata
 | 
true
| Polling Glue in background avoids query stalls. 
background_refresh_metadata_interval_millis
 | 
600000
| (10 min) Adjust based on how often Glue data changes. 
iceberg_metadata_memory_cache_capacity
 | 
2147483648
| (2GB) Increase for Iceberg manifest caching in FE memory. ────────── 3. CN / BE Configuration (
cn.conf
/ 
be.conf
) Optimize these for high-concurrency S3 scanning and local caching. Parameter | Recommended Value | Description 
storage_root_path
 | 
/path/to/fast/ssd
| CRITICAL: Enables the Data Cache. Caches S3 data on local SSD. 
pipeline_connector_scan_thread_num_per_cpu
 | 
16
or 
32
| Increases the number of parallel scan threads per core. 
max_hdfs_scanner_num
 | 
1024
| Increases the cap for concurrent remote scanners. 
hdfs_read_buffer_size_kb
 | 
16384
| (16MB) Larger buffers improve sequential read throughput from S3. ────────── 4. Catalog & Session Optimization These settings are applied at the Catalog level or via SQL sessions. Catalog Creation (Glue Specific) When creating your Glue catalog, ensure the region matches and caching is enabled:
Copy code
sql
CREATE EXTERNAL CATALOG glue_catalog
PROPERTIES (
    "type" = "hive", -- or "iceberg"
    "hive.metastore.type" = "glue",
    "aws.glue.region" = "us-east-1",
    "enable_metastore_cache" = "true",
    "metastore_cache_refresh_interval_sec" = "3600" -- Cache metadata for 1 hour
);
Session Variables for Large Queries Run these before querying "huge" tables to distribute the planning load: * 
SET GLOBAL plan_mode = 'distributed';
 Crucial for Iceberg/Hive. Allows CN nodes to help parse metadata (like Iceberg manifest files) so the FE doesn't OOM. * 
SET GLOBAL connector_io_tasks_per_scan_operator = 16;
 Increases I/O parallelism for S3 fetches. * 
SET GLOBAL enable_scan_datacache = true;
 Enables use of the local disk cache on CN nodes. Summary Checklist 1. FE RAM: Ensure 
-Xmx
is large enough to hold the Glue partition list. 2. Distributed Planning: Use 
plan_mode='distributed'
for metadata-heavy tables. 3. Data Cache: Use fast local NVMe/SSD on CN nodes to avoid repeated S3 I/O. 4. Network: Ensure StarRocks nodes are in the same region as the S3 bucket to avoid latency and data transfer costs.
Your CN configuration looks solid for a high-performance environment, but there are a few critical omissions and some nuances regarding the S3 remote spill and Data Cache setup that you should address. 1. Missing 
storage_root_path
(Critical for Data Cache) Even though you enabled 
datacache_enable = true
, you haven't specified where the data should be cached. By default, it goes to 
${STARROCKS_HOME}/storage
. For a "huge" database, you should point this to your high-speed local NVMe/SSD disks. * Recommendation: Add the following line:
Copy code
bash
    # Point to your NVMe/SSD mount points. Multiple paths separated by ';'
    storage_root_path = /data1/starrocks/storage;/data2/starrocks/storage
2. S3 Remote Spill Configuration (Incomplete) Your comment mentions "Disabled local storage and changed to S3 remote spill," but the configuration only points to a local directory. To actually enable spilling to S3 (which helps with massive joins/aggregations that exceed RAM), you need additional parameters: * Fix:
Copy code
bash
    # Enable remote spilling
    enable_spill_to_remote_storage = true
    # Specify the S3 path for spilling
    remote_spill_path = <s3://your-bucket/starrocks/spill/>

    # Note: spill_local_storage_dir is still used for small metadata/temp files
    # even when remote spill is enabled.
    spill_local_storage_dir = /opt/starrocks/cn/spill
3. BRPC Socket Buffer You set 
brpc_socket_max_unwritten_bytes
to 2GB (
2147483648
). While this is better than the 1GB default, for truly "huge" scale-out joins on AWS, StarRocks best practices often suggest moving to 4GB to completely eliminate "The server is overcrowded" errors during heavy shuffles. * Recommendation: 
brpc_socket_max_unwritten_bytes = 4294967296
4. Data Cache Block Size You set 
datacache_block_size = 262144
(256KB). * Context: This is the default. It works well for general cases. However, if your Glue data consists of very large Parquet/ORC files, a larger block size (e.g., 1MB) can sometimes improve sequential scan throughput, though 256KB is the safest starting point. Updated Review Summary Here is the refined version of your CN config with the missing pieces included:
Copy code
bash
# Server & Network
brpc_connection_type = pooled
brpc_socket_max_unwritten_bytes = 4294967296  # Increased to 4GB for huge shuffles

# Data Cache (Crucial for Glue/S3 performance)
datacache_enable = true
storage_root_path = /path/to/fast/nvme/storage  # MUST be defined for cache to work well
datacache_disk_size = 90%
datacache_mem_size = 30%
datacache_block_size = 262144

# Remote Spill to S3 (To handle massive query memory demand)
enable_spill_to_remote_storage = true
remote_spill_path = <s3://your-bucket-name/spill/>
spill_local_storage_dir = /opt/starrocks/cn/spill

# Additional optimization for high-throughput S3 scanning
pipeline_connector_scan_thread_num_per_cpu = 16
One Final Check: Ensure that your CN has the necessary IAM Role/Permissions to write to the 
remote_spill_path
S3 bucket. Unlike the Data Catalog (Glue) which is metadata, spilling involves writing large amounts of temporary data to S3. References • administration/management/BE_configuration.md • knowledge/usage/best-practice.md • administration/management/FE_configuration.md
The error log indicates a cascade failure caused by metadata and RPC timeouts between your Compute Nodes (CN) and Frontends (FE) in a shared-data (StarOS) environment. 1. Root Cause Analysis The logs point to three concurrent issues: * 
starlet err grpc.GetShard(...) Deadline Exceeded
: The CN tried to fetch shard/tablet metadata via the Starlet service (StarOS), but the request timed out. In shared-data mode, the FE manages this metadata. If the FE is too busy or under heavy Garbage Collection (GC), it cannot respond to these gRPC requests in time. * 
THRIFT_EAGAIN (timed out)
on port 9020: The CN is failing to report its execution state to the FE's Thrift port. This happens when the FE's RPC thread pool is exhausted or the FE is unresponsive. * 
Fail to wait EPOLLOUT
: This is a low-level network error. It means the CN's outgoing network buffer is full, and it cannot send more data because the receiver (FE or other CNs) isn't reading fast enough. In short: Your "huge database" has likely overloaded the FE's metadata management and RPC capacity, causing a bottleneck that times out the storage layer (Starlet). ────────── 2. How to Fix It Step 1: Increase FE Metadata & gRPC Timeouts Since the 
GetShard
request is failing, you need to give the StarOS manager (FE) more time to process requests under high load. In 
fe.conf
:
Copy code
properties
# Increase the gRPC timeout for StarOS metadata management (Default is often 5s)
starmgr_grpc_timeout_seconds = 30

# Increase the Thrift RPC timeout for status reporting
thrift_rpc_timeout_ms = 20000
Step 2: Solve FE Responsiveness (JVM & GC) With a "huge" database, the FE is likely suffering from long GC pauses. * Check FE Logs: Look at 
fe.gc.log
. If you see "Full GC" or pauses > 1s, you must increase memory. * Update 
fe.conf
:
Copy code
properties
    # Ensure you have at least 32GB-64GB for a large Glue/Shared-data catalog
    JAVA_OPTS="-Xmx64g -Xms64g ..."
Step 3: Fix CN Network Congestion (
EPOLLOUT
) The 
EPOLLOUT
error and 
THRIFT_EAGAIN
suggest the network pipes are "clogged." In 
cn.conf
:
Copy code
properties
# Use pooled connections to prevent head-of-line blocking on a single socket
brpc_connection_type = pooled

# Increase the socket buffer to 4GB to handle large metadata/data shuffles
brpc_socket_max_unwritten_bytes = 4294967296

# Increase the number of threads available to handle internal RPCs
brpc_num_threads = 64
Step 4: Optimize Starlet (CN Side) In 
cn.conf
, ensure the Starlet service (which handles the shared-data I/O) has enough resources:
Copy code
properties
# If using many tablets/shards, increase the metadata cache for Starlet
starlet_cache_thread_num = 32
Summary of Actions 1. FE: Increase 
starmgr_grpc_timeout_seconds
to 30. 2. FE: Bump 
JAVA_OPTS
to at least 64GB if you have millions of partitions. 3. CN: Set 
brpc_socket_max_unwritten_bytes
to 4GB. 4. Network: Ensure there is no significant latency between the CN and FE nodes (they should be in the same VPC/Subnet). References • faq/shared_data_faq.md • knowledge/admin_and_operation/shared_data_compaction.md
The error in your FE logs is related to a feature introduced in StarRocks 3.5.0 called Predicate Column Statistics. 1. Root Cause Analysis * 
PredicateColumnsDaemonThread
: This is a background thread that tracks which columns are used in 
WHERE
, 
JOIN
, or 
GROUP BY
clauses. It stores this metadata in the internal 
_statistics_.predicate_columns
table to optimize future statistics collection. * 
token recognition error at: '''
: This indicates that the internal SQL generator for this daemon thread encountered a character it could not parse—likely a single quote, backtick, or special character within a table name or column name that was not escaped properly. * 
StringBuilderEncoder.encode
Error: This is a secondary error from the logging framework (Log4j2). The daemon thread tried to log the failing query plan (the 
TPlanFragment
you see in the log), but the plan was so large that it exceeded Log4j's internal buffer, causing the encoder to crash while trying to report the original error. 2. How to Fix It The most effective way to stop this error and prevent potential FE instability is to disable the Predicate Column collection feature. Since this is a background optimization for the Cost-Based Optimizer (CBO), disabling it will not stop your queries from running, though it may slightly impact auto-statistics precision for very wide tables. Option A: Disable the feature (Recommended) Add or update this parameter in your 
fe.conf
and restart the FE:
Copy code
properties
# Disable predicate column collection to stop the background error
enable_predicate_columns_collection = false
_Note: You can also try setting this dynamically via 
ADMIN SET FRONTEND CONFIG ("enable_predicate_columns_collection" = "false");
but a config file change is safer for persistence._ Option B: Clean up the Statistics Table If you want to keep the feature but suspect the internal tracking table is corrupted or contains "bad" entries: 1. Check the status of the internal table:
Copy code
sql
    SELECT * FROM _statistics_.predicate_columns LIMIT 10;
2. If the table exists and contains suspicious characters in names, you may need to truncate it (StarRocks will recreate the entries as queries run):
Copy code
sql
    TRUNCATE TABLE _statistics_.predicate_columns;
Recommendation If you are on an early version of 3.5.x, this is likely a bug in the SQL generator for internal stats. Disable the feature using Option A to stabilize your FE, then consider upgrading to the latest patch release (e.g., 3.5.2+) where such parsing edge cases are typically patched. References • using_starrocks/Cost_based_optimizer.md • administration/management/FE_configuration.md References • administration/management/FE_configuration.md • using_starrocks/Cost_based_optimizer.md
In version 4.1.1, the error you are seeing is a known issue (often triggered by specific edge cases in SQL plans) where the background Predicate Column Statistics collector fails to parse internal metadata. 1. Root Cause * Internal SQL Bug: The 
PredicateColumnsDaemonThread
is responsible for tracking which columns are used in filters (
WHERE
, 
JOIN
, etc.) and saving them to the 
_statistics_.predicate_columns
table. The 
token recognition error at: '''
means the daemon is generating an internal SQL statement to update statistics but is failing to escape a single quote or special character in a table/column name. * Log Overflow: The 
StringBuilderEncoder.encode
error is a side effect. When the parser fails, StarRocks tries to log the entire query plan (the huge 
TPlanFragment
in your log) to help debug. This plan is so large that it exceeds the internal buffer of the Java logging framework (Log4j2), causing the "Recovering from StringBuilderEncoder" error. 2. How to Fix (Version 4.1.1) Immediate Workaround: Disable the Collection This is the most stable fix. Disabling this background thread will stop the errors and won't affect query results; it only means the CBO (Optimizer) will have slightly less "hinting" for which columns to prioritize for automatic stats. 1. Dynamic Fix (No Restart): Run this on your FE via any SQL client:
Copy code
sql
    ADMIN SET FRONTEND CONFIG ("enable_predicate_columns_collection" = "false");
2. Permanent Fix: Add the following line to your 
fe.conf
and restart the FE to ensure it persists:
Copy code
properties
    enable_predicate_columns_collection = false
Clean Up "Bad" Metadata If the daemon is stuck trying to process an existing "bad" record in the statistics table, you can clear the recorded predicate history:
Copy code
sql
TRUNCATE TABLE _statistics_.predicate_columns;
3. Summary for 4.1.1 This behavior is generally harmless to your data but can fill up FE logs and cause minor CPU spikes in the FE. Because version 4.1.1 is a very recent release, this specific parsing bug is likely slated for a fix in a future patch (e.g., 4.1.2 or 4.2). Until then, keeping 
enable_predicate_columns_collection
set to 
false
is the recommended path for production stability. Check FE Health: After disabling the parameter, monitor 
fe.log
to ensure the 
PredicateColumnsDaemonThread
has stopped and that your FE memory usage (JVM Heap) stabilizes. References • administration/management/FE_configuration.md • using_starrocks/Cost_based_optimizer.md References • administration/management/FE_configuration.md • using_starrocks/Cost_based_optimizer.md • administration/management/BE_configuration.md
To troubleshoot high memory usage on FE and CN/BE nodes in StarRocks, follow this runbook. 1. Frontend (FE) Memory Runbook FE memory is primarily managed by the JVM Heap. High usage usually stems from massive metadata, high query concurrency, or long GC pauses. Step 1: Check JVM Heap & GC Status * Command: 
jstat -gcutil <fe_pid> 1000 10
* What to look for: If 
O
(Old Generation) is near 100% and 
FGC
(Full GC) count is increasing, the FE is struggling with a lack of heap space. * Action: Increase 
JAVA_OPTS="-Xmx"
in 
fe.conf
. For large clusters, 32GB–64GB is standard. Step 2: Identify Memory Consumers (StarRocks 3.3.6+) StarRocks now automatically dumps memory profiles when a surge occurs. * Location: 
fe/log/proc_profile/*.tgz
* Action: Decompress the latest HTML file and open it in a browser. Look for the "Flame Graph" to see which module (e.g., 
GlobalStateMgr
, 
QueryCoordinator
) is consuming the most memory. Step 3: Track Leaks (StarRocks 3.3.7+) Check 
fe.log
for Memory Usage Tracker entries. It logs the "Shadow Memory" of internal modules. * Key Modules: * `LocalMetastore`: Table/Tablet metadata. * `QueryCoordinator`: Active query execution structures. * `EditLog`: Metadata change logs waiting to be checkpointed. Step 4: Analyze Object Histogram If heap is full, run: 
jmap -histo:live <fe_pid> | head -n 20
. * If 
com.starrocks.catalog.Tablet
or 
Replica
is at the top, you have too many tablets. Consolidate partitions or increase the FE heap. ────────── 2. Compute Node (CN) / Backend (BE) Memory Runbook CN/BE memory is C++ based and managed by internal MemTrackers. Step 1: View Global Memory Breakdown The most critical tool is the 
mem_tracker
endpoint: * Command: 
curl -XGET http://<CN_IP>:<CN_HTTP_PORT>/mem_tracker
* Key Categories to check: * 
query
: Memory used by currently running queries. If high, check 
SHOW PROC '/current_queries'
. * 
storage_page_cache
: Data cache. If high, reduce 
storage_page_cache_limit
in 
cn.conf
. * 
metadata
: Cached tablet/segment metadata. * 
process
: Total memory seen by the OS. Step 2: Check Node Status * Command: 
SHOW PROC '/backends'\G
or 
SHOW PROC '/compute_nodes'\G
. * Metric: 
MemUsedPct
. If it's consistently > 80%, the node will start "soft-limit" cancellations to protect itself. Step 3: Identify Expensive Queries If 
query
memory is high in Step 1: 1. Run 
SHOW PROC '/current_queries'
to find queries with high 
MemUsage
. 2. Use 
KILL QUERY <query_id>
to free memory immediately. 3. Adjust 
exec_mem_limit
(default is often too high or unset) to cap individual queries. Step 4: Check for TCMalloc Overhead Sometimes physical memory is high but MemTrackers show low usage. This is "memory fragmentation." * Command: 
curl -XGET http://<CN_IP>:<CN_HTTP_PORT>/metrics | grep tcmalloc
* Metric: 
starrocks_be_tcmalloc_bytes_in_use
vs 
starrocks_be_tcmalloc_pageheap_free_bytes
. * Action: If 
pageheap_free_bytes
is huge, TCMalloc isn't returning memory to the OS. Try setting 
tc_use_memory_minimized = true
in 
cn.conf
. Quick Mitigation Summary Problem | Action FE Full GC | Increase 
Xmx
in `fe.conf`; enable 
MALLOC_ARENA_MAX=1
. CN Metadata High | Increase CN count or reduce partition/tablet count. CN Query OOM | Set 
exec_mem_limit
session variable to a lower value (e.g., 
8G
). CN Page Cache High | Lower 
storage_page_cache_limit
(default is 20% of total mem). References • faq/fe_mem_faq.md • administration/management/BE_configuration.md • faq/Sql_faq.md References • faq/Sql_faq.md
To enable debug logging in StarRocks, you can use either static configuration (requires a restart) or dynamic adjustment (takes effect immediately without a restart). 1. Frontend (FE) Logging The FE uses Log4j. Note that "DEBUG" level can be very chatty, so it is often better to enable it for specific modules. Dynamic (No Restart) The most flexible way to enable FE debug logs at runtime is via the FE's HTTP interface. * Enable for a specific package:
Copy code
bash
    curl -X POST "http://{fe_ip}:{fe_http_port}/log?add_verbose=org.apache.starrocks.sql.optimizer"
* Disable/Reset verbose logging:
Copy code
bash
    curl -X POST "http://{fe_ip}:{fe_http_port}/log?del_verbose=org.apache.starrocks.sql.optimizer"
* View current log levels: Access 
http://{fe_ip}:{fe_http_port}/log
in your browser. Static (Permanent) Modify 
fe/conf/fe.conf
and restart the FE:
Copy code
properties
# Change global level (not recommended for production)
sys_log_level = DEBUG

# OR enable for specific modules (recommended)
sys_log_verbose_modules = org.apache.starrocks.catalog, org.apache.starrocks.sql.optimizer
────────── 2. Compute Node (CN) / Backend (BE) Logging CN/BE debug logs are managed via 
VLOG
(verbose logs). These logs appear in your 
cn.INFO.log
or 
be.INFO.log
. Dynamic (No Restart) You can update the configuration via SQL or the HTTP API. * Via SQL (Recommended for v3.1+):
Copy code
sql
    -- Enable verbose logs for all modules (level 10 is standard)
    UPDATE information_schema.be_configs SET VALUE = "*" WHERE NAME = "sys_log_verbose_modules";
    UPDATE information_schema.be_configs SET VALUE = "10" WHERE NAME = "sys_log_verbose_level";
* Via HTTP API:
Copy code
bash
    curl -X POST "http://{cn_ip}:{cn_http_port}/api/update_config?sys_log_verbose_modules=*"
    curl -X POST "http://{cn_ip}:{cn_http_port}/api/update_config?sys_log_verbose_level=10"
Static (Permanent) Modify 
cn/conf/cn.conf
(or 
be.conf
) and restart:
Copy code
properties
# Enable for specific C++ files or wildcards
sys_log_verbose_modules = *
# Controls the depth of the debug info (1-10)
sys_log_verbose_level = 10
3. Summary of Log Levels * INFO: Standard operational logs. * DEBUG/VLOG: Detailed execution steps (use only for troubleshooting). * WARN/ERROR: Issues that need attention. Caution: Enabling 
*
(all modules) at 
DEBUG
level or 
sys_log_verbose_level = 10
on a high-concurrency production cluster can cause logs to rotate extremely fast and may impact disk I/O performance. Always remember to revert the settings after troubleshooting. References • administration/management/logs.md • administration/management/FE_configuration.md • administration/management/BE_configuration.md References • administration/management/FE_configuration.md • administration/management/BE_configuration.md
This log message occurs during the startup process of a Frontend (FE) node. It indicates that the startup script tried to find an existing cluster leader to join but failed to connect to any other FE nodes within the timeout period. Depending on your situation, this is either normal or a configuration error. 1. When this is NORMAL * Initial Deployment: If this is the very first FE node you are starting in a new cluster, this message is expected. Since no other FE exists yet, the node correctly "assumes" it is the first node and initializes the cluster metadata. * Cluster Recovery: If you are restarting the entire cluster from scratch and this is the first node coming back online. 2. When this is an ERROR If you already have a running cluster and are trying to add a new Follower or Observer FE, this message indicates a discovery failure. Instead of joining your existing cluster, this node might try to start a new, independent cluster (which will fail or lead to a split-brain). Troubleshooting Steps If this is not your first FE node, check the following: * Helper Address: Verify the 
--helper
parameter in your startup command (or the `FE_SERVERS`/`meta_helper` setting in K8s). Ensure it points to the correct IP/hostname and Edit Log Port (default 
9010
) of the existing Leader FE. * Network Connectivity: Ensure the new node can reach the existing Leader FE on: * 
9010
(Edit Log Port/BDBJE) * 
9020
(RPC Port) * 
9030
(Query Port) * DNS Resolution: If using hostnames (common in Kubernetes), ensure the new node can resolve the helper's hostname. Check for errors like 
Temporary failure in name resolution
in the logs. * Firewalls/Security Groups: Ensure there are no firewall rules blocking traffic between FE nodes on the ports mentioned above. * Existing Leader Status: Verify the existing cluster is healthy. Run 
SHOW FRONTENDS;
on the running leader to confirm it is actually "Alive". 3. How to fix a "Stuck" node If the node mistakenly started as a "standalone" leader and now cannot join the real cluster: 1. Stop the faulty FE process. 2. Clear Metadata: Delete the contents of the 
meta/
directory on the new node (defined by 
meta_dir
in 
fe.conf
) to wipe the incorrect standalone metadata. 3. Add Node to Cluster: On the existing Leader, run:
Copy code
sql
    ALTER SYSTEM ADD FOLLOWER "IP_of_new_node:9010";
4. Restart: Start the new FE again using the correct 
--helper
pointing to the Leader. Note: If you are using the StarRocks Kubernetes Operator, this message often appears if the 
starrocks-fe-0
pod is still initializing or if the Service naming doesn't match the configuration. Check the 
Ready
status of your other FE pods. References • administration/management/logs.md • deployment/deploy_manually.md • administration/management/FE_configuration.md
Recovering a StarRocks Frontend (FE) depends on the severity of the failure. Follow the scenario that matches your situation. Scenario 1: A single Follower/Observer failed (Leader is Alive) If you still have a working Leader FE, this is the safest and most common recovery. 1. Drop the node from the Leader:
Copy code
sql
    -- Connect to the Leader FE and run:
    ALTER SYSTEM DROP FOLLOWER "failed_fe_ip:9010";
    -- (Or DROP OBSERVER if it was an observer)
2. Clean up: Stop the failed FE process and delete the contents of its 
meta_dir
(defined in 
fe.conf
). 3. Re-add the node:
Copy code
sql
    ALTER SYSTEM ADD FOLLOWER "failed_fe_ip:9010";
4. Restart with Helper:
Copy code
bash
    ./bin/start_fe.sh --helper leader_ip:9010 --daemon
────────── Scenario 2: Disaster Recovery (Majority Lost / No Leader) Use this if the majority of your FEs are down and they cannot elect a leader (e.g., 2 out of 3 FEs are dead). 1. Identify the node with the latest data: On each FE server, run the BDBJE utility to check the `lastVLSN`:
Copy code
bash
    java -jar fe/lib/starrocks-bdb-je-18.3.16.jar DbPrintLog -h fe/meta/bdb/ -vd | grep lastVLSN
The node with the largest 
lastVLSN
 is your best recovery candidate. 2. Reset the Election Group: On that candidate node, edit 
fe.conf
and add:
Copy code
properties
    bdbje_reset_election_group = true
3. Start the node: It will force itself to become the Leader. 4. Cleanup: Once it is up and 
SHOW FRONTENDS
shows it as 
LEADER
, remove the 
bdbje_reset_election_group
line from 
fe.conf
(otherwise it will reset every time it restarts). 5. Rebuild others: Follow Scenario 1 to clear and re-add the other failed FEs. ────────── Scenario 3: Metadata Corruption If logs show errors like 
catch exception when replaying
or `failed to load journal type`: 1. Skip Bad Journals (if you know the ID): Add this to 
fe.conf
and restart:
Copy code
properties
    metadata_journal_skip_bad_journal_ids = 12345,12346
2. Metadata Recovery Mode (Last Resort): If the corruption is widespread, add this to `fe.conf`:
Copy code
properties
    metadata_enable_recovery_mode = true
Note: This starts the FE in a read-only state to attempt a repair. Once up, run 
ALTER SYSTEM CREATE IMAGE;
to save a clean state, then remove the config and restart. ────────── Scenario 4: FE IP Address Changed If an FE server's IP changed, it will fail to join the BDBJE group because the metadata stores the old IP. 1. Check the expected IP in 
fe/meta/image/ROLE
. 2. Ensure 
priority_networks
in 
fe.conf
is set to the correct subnet to force the FE to bind to the right IP. 3. If the IP must be new, follow the Scenario 1 steps (Drop, Clear Meta, Add with Helper). Important Safety Tips: * Always back up your 
meta/
directory before attempting any manual recovery. * Check 
fe/log/fe.log
constantly during recovery to see if the node is replaying journals or stuck in an election. References • administration/Meta_recovery.md • administration/management/FE_configuration.md • deployment/deploy_manually.md References • administration/Meta_recovery.md • administration/management/FE_configuration.md
2 Views
Open in Slack
PreviousNext
Powered by