http://starrocks.io logo
Join Slack
Powered by
Hi Rocky <@U0A71G31CDV> Our Environment: StarRoc...
# questions-and-troubleshooting
r
Hi Rocky @Rocky Our Environment: StarRocks 4.0.4 on Kubernetes, deployed via StarRocks Operator. Storage: S3 + Hive Metastore (HMS). Query engine uses External Catalog (Iceberg tables). CN already has enable_spill=true. We frequently experience OOM on CN Pods. We already have enable_spill=true enabled. What additional options should we configure to prevent CN OOM?
r
Hi there! Rocky here. Dealing with OOM (Out of Memory) on CN pods in a Kubernetes environment while querying Iceberg tables can be tricky, even with spilling enabled. Since you're on version 4.0.4, here are the key configuration options and architectural adjustments you should check to prevent CN OOM: 1. Align Pod Memory Limits with StarRocks 
mem_limit
 In Kubernetes, if your pod memory limit is reached before StarRocks' internal memory management triggers, the kernel OOM killer will terminate the pod. * The Problem: By default, 
mem_limit
is 90%. In a pod with 64GB RAM, StarRocks thinks it can use ~57GB. However, factors like TcMalloc overhead, metadata caches, and OS overhead often push actual usage over 64GB before StarRocks hits that 90% threshold. * The Fix: Lower 
mem_limit
in your 
cn.conf
(or via the StarRocks Operator 
config
section) to 80% or 85%.
Copy code
properties
    mem_limit = 80%
2. Verify and Optimize Spill Configurations Simply setting 
enable_spill = true
isn't always enough if the triggering thresholds or storage paths aren't optimized. * Check `spill_local_storage_dir`: Ensure this is explicitly set in 
cn.conf
and points to a persistent volume with high IOPS (preferably SSD). If it's not set, spilling might fail or use a slow/small root partition. * Lower the Spill Threshold: By default, spilling starts when memory usage hits 100% of the query pool (
query_pool_spill_mem_limit_threshold = 1.0
). To prevent OOM, trigger it earlier (e.g., at 80%):
Copy code
properties
    # In cn.conf (requires restart)
    query_pool_spill_mem_limit_threshold = 0.8
* Enable Sink Spilling (if performing 
INSERT INTO
): Since you are on v4.0, if you are writing data back to Iceberg, ensure sink spilling is enabled:
Copy code
sql
    SET GLOBAL enable_connector_sink_spill = true;
3. Tune Iceberg/External Scan Memory Usage External scans (S3 + HMS) can consume significant memory for metadata and I/O buffers. * Limit Metadata Cache: By default, StarRocks allocates a percentage of memory for Iceberg metadata. You can restrict this if your tables have millions of files: * 
iceberg_data_file_cache_memory_usage_ratio
(Default 0.1) * 
iceberg_delete_file_cache_memory_usage_ratio
(Default 0.1) * Control I/O Concurrency: If too many concurrent I/O tasks are running, memory usage spikes. Try reducing the tasks per operator:
Copy code
sql
    SET GLOBAL connector_io_tasks_per_scan_operator = 8; -- Default is 16
4. Implement Query-Level Safeguards To prevent a single "rogue" query from eating all memory and killing the CN: * Set `query_mem_limit`: Define a hard limit for any single query on a node.
Copy code
sql
    SET GLOBAL query_mem_limit = 16G; -- Adjust based on your Pod size
* Use Resource Groups: Create a resource group for External Catalog queries to cap their total memory usage:
Copy code
sql
    CREATE RESOURCE GROUP iceberg_queries
    PROPERTIES (
        "mem_limit" = "60%",
        "concurrency_limit" = "10"
    );
5. Troubleshooting Tips * Check 
<http://be.INFO|be.INFO>
(on CN): Look for 
Memory Limit Exceeded
logs. It will show a "Memory Tracker" breakdown indicating which component (e.g., 
ScanNode
, 
IntermediateResult
, 
Metadata
) is consuming the most memory. * K8S Events: Run 
kubectl describe pod <cn-pod>
to see if it was indeed 
OOMKilled
(Exit Code 137). If it was, focusing on Point #1 (Memory Alignment) is your priority. Summary Recommendation: Start by setting 
mem_limit = 80%
and 
query_pool_spill_mem_limit_threshold = 0.8
. This forces StarRocks to be more conservative and start spilling before the Kubernetes Pod limit is reached its hard limits. References • administration/management/BE_configuration.md
r
@Rocky Offloading FE query planning to CN. We want to speed up the FE query planning phase, especially for Iceberg external catalog queries. Is it a good strategy to offload planning work to CN? - Is SET GLOBAL plan_mode = 'distributed' the recommended approach to delegate Iceberg manifest parsing to CN? - Any caveats or trade-offs of plan_mode=distributed vs auto vs local? - Are there other FE planning bottlenecks that can be offloaded, such as statistics collection or metadata parsing? - What are the recommended values for iceberg_metadata_memory_cache_capacity and iceberg_manifest_cache options when using plan_mode=distributed?
r
Offloading FE query planning to CN nodes is an excellent strategy for large-scale Iceberg tables where the FE might become a bottleneck due to the sheer volume of manifest files. Since you are on StarRocks 4.0.4, here are the answers to your specific questions: 1. Is 
SET GLOBAL plan_mode = 'distributed'
the recommended approach? Yes. Setting 
plan_mode = 'distributed'
is the direct way to delegate Iceberg manifest parsing and metadata retrieval to CN nodes. * How it works: Instead of the FE reading, decompressing, and parsing every manifest file, it generates a "mini-plan" for the CN nodes. The CN nodes then process these manifest files in parallel, filtering out irrelevant data files before sending the pruned metadata back to the FE for the final physical plan generation. * When to use: It is highly recommended for queries against Iceberg tables with thousands of manifest files or very large numbers of data files. 2. Caveats and Trade-offs of 
plan_mode
 Mode | Strategy | Best For | Trade-offs 
local
 | FE parses all manifest files and caches deserialized objects. | Small/Medium metadata, high-frequency repeated queries. | Can cause FE OOM or high CPU/GC pauses if metadata is huge. 
distributed
 | CN nodes parse manifest files in parallel. | Large metadata, big tables with many partitions/files. | Small overhead for scheduling tasks; slightly higher latency for very small queries. 
auto
 (Default) | Automatically chooses based on cluster resources and manifest count. | Most general production workloads. | Sometimes 
auto
might stay 
local
too long; manual override to 
distributed
is safer for "huge" tables. 3. Other FE Planning Bottlenecks & Offloading Beyond manifest parsing, you can offload or optimize the following: * Statistics Collection: In v4.0, statistics collection (
ANALYZE
) is already executed as a distributed job on CN nodes. Ensure 
enable_query_trigger_analyze = true
is set (it is by default in v3.4+) so StarRocks automatically collects stats for your Iceberg columns in the background. * Column Stats Pruning: You can enable 
SET GLOBAL enable_iceberg_column_statistics = true;
. This allows the distributed planner to use min/max/null-count from Iceberg manifests to prune even more files, though it increases the amount of metadata processed. * Metadata Caching: Enable the Iceberg Metadata Disk Cache on the FE to reduce S3/HMS calls:
Copy code
sql
    -- FE configuration (requires restart or admin set)
    enable_iceberg_metadata_disk_cache = true
    iceberg_metadata_memory_cache_capacity = 2147483648 -- Increase to 2GB if FE has RAM
4. Recommended Values for 
plan_mode = distributed
 When using distributed planning, the memory pressure shifts from the FE's "deserialized object cache" to the CNs' compute and buffer memory. * 
iceberg_metadata_memory_cache_capacity
(FE Config): Even with 
distributed
mode, the FE needs to cache the "Snapshot" and "Manifest List" (the list of manifests). * Recommendation: Keep it at 1GB - 2GB if you have many catalogs. If your FE is memory-constrained, you don't need to set this very high because the heavy "Manifest" data is handled by CNs. * 
iceberg_manifest_cache_with_column_statistics
(Catalog Property): This controls whether to include column-level stats (min/max) in the cache. * Recommendation: Set to 
true
if you want better pruning, but be aware it increases the memory footprint of the metadata. * CN-side Memory (Important): Since your CNs are already experiencing OOM, distributed planning will add some additional memory pressure. Ensure your 
mem_limit
is aligned (as discussed in the previous response) and monitor the 
ConnectorScanNode
in your query profiles. Summary Strategy 1. Set 
plan_mode = 'distributed'
 globally. 2. Enable the Disk Cache on FE to persist metadata across restarts.
3. Set 
enable_iceberg_column_statistics = true
 to improve planning accuracy at the cost of slight CN memory increase. memory usage. 4. Monitor FE/CN logs specifically for 
Metadata Cache
hit rates to see if your capacity settings are sufficient. References • data_source/catalog/iceberg/iceberg_catalog.md • data_source/feature-support-data-lake-analytics.md
r
@Rocky Last Question. StarRocks + S3 + HMS + Iceberg External Catalog optimization. Our stack is StarRocks 4.0.4 as the query engine, data on S3, metadata on Hive Metastore (HMS), and tables accessed via Iceberg External Catalog. What other optimization options are worth tuning? Specifically: - Data Cache (Block Cache) sizing and hit-rate tuning for S3 reads - Iceberg-specific tuning such as manifest cache, metadata cache, predicate pushdown, and partition pruning - HMS connection pooling and metadata refresh strategy - S3 I/O tuning such as object_storage_connect_timeout_ms, object_storage_request_timeout_ms, and connection pool size - Any recommended FE/CN config differences for external-catalog-heavy workloads versus internal table workloads - Query profile and slow-query debugging tips specific to Iceberg external catalog
r
For a high-performance StarRocks 4.0.4 environment using S3, HMS, and Iceberg, the following optimizations are categorized by layer. 1. Data Cache (Block Cache) Optimization This is the most critical feature for reducing S3 read latency and egress costs. In StarRocks 4.x, the Data Cache splits remote files into blocks and stores them on local disks of the CN/BE nodes. * CN Configuration (
cn.conf
): * `datacache_enable = true`: Enable the feature. * `datacache_mem_size = 10%`: Portion of memory used for metadata and indexing of the cache. * `datacache_disk_path = /path/to/fast/ssd; /path/to/another/ssd`: Crucial on K8s. Ensure you mount fast NVMe/SSD storage (using HostPath or Local PVs) to these paths. * `datacache_disk_size = 500G`: Specify the size per disk path. * Session Variable: * `SET GLOBAL enable_scan_datacache = true;`: Required to actually trigger cache usage during scans. * Tuning Hit-Rate: * Use 
EXPLAIN ANALYZE
or the query profile. Look for 
DataCacheReadBytes
vs 
BytesRead
. If the hit rate is low, increase the disk size or evaluate if your working set fits in the allocated cache. 2. Iceberg-Specific Tuning Beyond 
plan_mode = distributed
, fine-tune how StarRocks handles Iceberg metadata. * Metadata Disk Cache (FE side): StarRocks can cache Iceberg metadata files on the FE disk to speed up restarts and planning.
Copy code
properties
    # fe.conf
    enable_iceberg_metadata_disk_cache = true
    iceberg_metadata_disk_cache_capacity = 10737418240  # 10GB
* Column Statistics Pruning: By default, StarRocks might not use Iceberg column stats (min/max/nulls) for data file pruning unless explicitly enabled. This significantly reduces the number of files scanned.
Copy code
sql
    SET GLOBAL enable_iceberg_column_statistics = true;
* Predicate Pushdown: Ensure 
enable_connector_adaptive_scan_concurrency = true
(default). This allows StarRocks to adjust the number of scan threads based on the data volume and system load. 3. HMS & Metadata Refresh Strategy HMS can become a bottleneck when many partitions or tables are accessed frequently. * Connection Pooling: Increase the HMS client pool size if you see "Waiting for HMS connection" in FE logs:
Copy code
properties
    # fe.conf
    metastore_agent_max_connections = 32
* Refresh Strategy: By default, StarRocks checks for metadata changes every few minutes. * High Freshness: If you need near-real-time updates, set the catalog property 
"metastore_cache_refresh_interval_sec" = "60"
. * High Performance: If data changes only once a day, set this to a higher value (e.g., 
3600
) to avoid frequent HMS polling. * Manual Refresh: For specific tables, use 
REFRESH EXTERNAL TABLE table_name;
. 4. S3 I/O Tuning When querying S3, you want high concurrency and resilience against transient network blips. * Timeouts & Connections (
cn.conf
):
Copy code
properties
    # Default is often -1 (SDK default). Tighten if you see hung queries.
    object_storage_connect_timeout_ms = 2000
    object_storage_request_timeout_ms = 10000

    # Increase the S3 connection pool if you have high-core CN nodes
    object_storage_max_connection = 1024
5. FE/CN Config: Internal vs External Focus External catalog queries behave differently than internal table queries. * CN Memory Management: As discussed earlier, set 
mem_limit = 80%
. External scans use large memory buffers for Parquet/ORC decompression. * I/O Concurrency:
Copy code
sql
    SET GLOBAL connector_io_tasks_per_scan_operator = 16;
If your CN pods are CPU-bound, decrease this. If they are idling while waiting for S3, increase it. 6. Debugging Iceberg Performance To find out why a query is slow, look for these specific indicators in the Query Profile:
1. 
IcebergTableTimer
: Time spent interacting with HMS/S3 for metadata. If high, check 
plan_mode
and HMS latency. 2. 
ScanTime
: Total time spent reading data. 3. 
IOTaskWaitTime
: If this is high, it means scan threads are waiting for an I/O slot. Increase 
connector_io_tasks_per_scan_operator
. 4. 
DataCacheReadBytes
: If this is 0, your Data Cache is not working or not enabled for that query. 5. 
RowsRead
vs 
RawRowsRead
: If 
RawRowsRead
is much higher, your partition pruning or predicate pushdown isn't working effectively. Check your 
WHERE
clauses and column stats. Rocky's Pro-Tip: On Kubernetes, always monitor the 
sar -n DEV
or K8s network metrics on the node. High S3 throughput can easily saturate a 10Gbps or even 25Gbps network interface if many CN pods are running on the same physical host! References • best_practices/query_tuning/query_profile_operator_metrics.md • loading/loading_introduction/troubleshooting_loading.md
r
@Rocky Thanks Rocky. Follow-up question. For the options you recommended (mem_limit, spill threshold, plan_mode=distributed, Data Cache, Iceberg metadata cache, HMS pool, S3 timeouts, etc.), how do we verify each option is actually working? For each option, please tell us: - Which SQL command, log file, or query profile field to check - What value or output indicates it is working correctly - A simple test scenario if possible We want to build a verification checklist before rolling these to production.
r
This is a great approach. A verification checklist ensures your "optimizations" are actually being applied by the engine. Below is the verification guide for StarRocks 4.0.4 on K8s. 1. Memory & Spilling (OOM Prevention) * Verification Method: Query Profile (
SET pipeline_profile_level = 1;
then 
ANALYZE PROFILE FROM 'query_id';
) or CN Logs (
<http://be.INFO|be.INFO>
). * Indicator of Success: * Profile: Look for 
QuerySpillBytes
in the Execution Summary. If it is 
> 0
, spilling is working. * Profile: Look for 
QueryPeakMemoryUsagePerNode
. It should be below your pod's hard limit (e.g., if pod is 64G and 
mem_limit=80%
, this should peak around 51G). * CN Log: Grep for 
schema_change_mem_limit
or 
query_mem_limit
to see if the configuration was loaded. * Simple Test:
Copy code
sql
    SET enable_spill = true;
    SET spill_mode = 'force'; -- Forces spilling even if memory is plenty
    SELECT count(distinct large_col) FROM iceberg_table; -- Large aggregation
Check the profile for 
QuerySpillBytes > 0
. 2. Distributed Planning (
plan_mode = distributed
) * Verification Method: 
EXPLAIN
command or Query Profile. * Indicator of Success: * Explain: Run 
EXPLAIN SELECT ...
. In distributed mode, you will see multiple Fragments dedicated to metadata scanning (e.g., 
IcebergMetadataScan
). * Profile: Look for 
IcebergTableTimer
or 
MetadataScanTime
. In 
distributed
mode, these tasks will show up as being executed across multiple CN nodes rather than just the FE. * Simple Test:
Copy code
sql
    SET plan_mode = 'distributed';
    EXPLAIN SELECT * FROM your_huge_iceberg_table LIMIT 1;
Verify the plan has more than 1 fragment and includes a 
ConnectorScanNode
for metadata. 3. Data Cache (S3/Block Cache) * Verification Method: Query Profile + 
EXPLAIN VERBOSE
. * Indicator of Success: * Explain Verbose: Look for 
dataCacheOptions={populate: true}
in the scan node description. * Profile: Look for 
DataCacheReadBytes
and 
DataCacheWriteBytes
. * Working: First run shows 
DataCacheWriteBytes > 0
. * Hitting: Second run shows 
DataCacheReadBytes > 0
and 
DataCacheWriteBytes = 0
. * Simple Test: Run the same query twice on a table:
Copy code
sql
    SELECT count(*) FROM iceberg_table WHERE partition_col = '2023-01-01';
If successful, the 
DataCacheReadBytes
in the second profile should equal the data volume of that partition. 4. Iceberg Metadata & Manifest Cache * Verification Method: FE Logs (
fe.log
) or 
SHOW PROC
. * Indicator of Success: * FE Log: Grep for 
Iceberg metadata cache hit
or 
Iceberg manifest cache
. * Proc: 
SHOW PROC '/catalogs';
(This shows some metadata cache statistics in newer versions). * Simple Test: Execute a query that requires partition pruning. Repeat it 3 times. The "Planning Time" in the FE Summary (visible in 
SHOW PROFILELIST
) should drop significantly after the first run. 5. S3 I/O & Connection Tuning * Verification Method: CN Configuration check (
SHOW CONFIG
) and CN Logs. * Indicator of Success: * SQL: 
SHOW CONFIG LIKE '%object_storage%';
— Verify the values match your 
cn.conf
. * Profile: Look for 
IOTaskWaitTime
. If this value is consistently high (e.g., > 500ms), your 
object_storage_max_connection
or 
connector_io_tasks_per_scan_operator
might still be too low. * Simple Test: Run a high-concurrency scan. If you see 
S3 Service Unavailable
or 
Timeout
in logs, your connection pool is exhausted or timeouts are too aggressive. 6. HMS Connection Pooling * Verification Method: FE Logs. * Indicator of Success: * FE Log: Grep for 
MetastoreClientPool
. * Metric: If you have Prometheus/Grafana, monitor the metric 
starrocks_fe_hms_connection_pool_size
. * Simple Test:
Run 20+ concurrent queries that touch different Iceberg tables. If they all start planning immediately without "waiting for HMS connection" logs, your pool is sized correctly. Summary Verification SQL Checklist Run these commands after your CN pods/FE are restarted:
Copy code
sql
-- 1. Check CN Runtime Config
ADMIN SHOW FRONTEND CONFIG LIKE '%iceberg%';
SHOW CONFIG LIKE '%mem_limit%';
SHOW CONFIG LIKE '%datacache%';

-- 2. Check Session Variables
SHOW VARIABLES LIKE '%spill%';
SHOW VARIABLES LIKE 'plan_mode';

-- 3. Run Test Query and Analyze
SET pipeline_profile_level = 1;
<YOUR_QUERY>;
SHOW PROFILELIST; -- Get the Query ID
ANALYZE PROFILE FROM '<QUERY_ID>';
Rocky's Tip: For Kubernetes, ensure your Data Cache path (
datacache_disk_path
) is actually mounted to a high-speed volume. If it's accidentally pointing to the pod's ephemeral storage, it might be throttled or could cause the pod to get evicted if it fills up the root disk! References • administration/management/resource_management/spill_to_disk.md • best_practices/query_tuning/query_profile_operator_metrics.md
r
@Rocky One last question Rocky. When creating a StarRocks cluster via CustomResource (StarRocks Kubernetes Operator v1.11.0), what is the best practice for applying FE-level global settings that need to survive restarts? For example, settings like: - plan_mode = distributed - enable_iceberg_column_statistics = true - connector_io_tasks_per_scan_operator = 8 These are SET GLOBAL variables, but SET GLOBAL is volatile and resets when FE restarts. How do you make them permanent? I can think of several possible options. Which one is best, and why? Option 1. PreStart hook or init script in the FE Pod that runs SET GLOBAL via mysql client after FE is ready. Option 2. Writing them into fe.conf (or cn.conf) via the Operator ConfigMap. Option 3. Using a Kubernetes Job or sidecar that applies the settings once after cluster startup. Option 4. Defining them inside the StarRocksCluster CR fields directly, if the Operator supports a config injection field. Option 5. Using a custom Operator reconciler or a GitOps tool (ArgoCD) to apply SQL after cluster becomes ready. Specifically please answer: - Which SET GLOBAL variables are actually persistable via fe.conf, and which are session-only and must be re-applied every restart? - Does the StarRocks Kubernetes Operator v1.11.0 have an official field for injecting SQL init statements or conf overrides per cluster? - For the unpersistable ones, what is the most production-grade pattern? PreStart script, Job, sidecar, or something else? - Any safety concerns with preStart scripts racing with FE readiness? We want a pattern that is idempotent, survives FE pod restart, and works cleanly with the StarRocks Operator.
r
Managing global settings in a Kubernetes environment requires distinguishing between Configuration Parameters (fe.conf) and System Variables (SET GLOBAL). Here is the breakdown of how to handle these in production with StarRocks Operator v1.11.0. 1. Persistence Truths: Metadata vs. Conf Files * System Variables (
SET GLOBAL
): In StarRocks, variables like 
plan_mode
and 
enable_iceberg_column_statistics
are stored in the FE's internal Metadata (Raft log). As long as your FE pods use Persistent Volumes (PVs) for their metadata directory (the default in production), these settings survive FE pod restarts. You only need to run the command once in the cluster's lifetime. * FE Configuration (
fe.conf
): Parameters viewable via 
ADMIN SHOW FRONTEND CONFIG
(like 
http_port
or 
metastore_agent_max_connections
) must be in 
fe.conf
. If you set them via 
ADMIN SET FRONTEND CONFIG
, they are volatile and reset on restart unless written to the file. 2. Best Practice: The Kubernetes "Init Job" Pattern Since the StarRocks Operator v1.11.0 does not have an official 
initSql
field in the CRD, the most production-grade and idempotent approach is a Kubernetes Job that runs after the cluster is deployed. Why Option 3 (Job) is better than others: * Option 1 (PreStart/PostStart): Often fails because the FE process is "running" but the MySQL protocol isn't "ready" to accept connections yet. * Option 2 (ConfigMap): You cannot put session variables like 
plan_mode
in 
fe.conf
. StarRocks will ignore them or fail to start. * Option 3 (Job): Can include a 
wait-for-it
logic to ensure the FE is fully healthy before executing SQL. 3. Implementation Checklist for Production Step A: Apply Persistent FE Config (
fe.conf
) For variables that can go in the config file (like HMS pool size or S3 timeouts), use the 
config
block in your 
StarRocksCluster
CR. The Operator will automatically inject these into the generated 
fe.conf
and 
cn.conf
.
Copy code
yaml
apiVersion: <http://starrocks.com/v1|starrocks.com/v1>
kind: StarRocksCluster
spec:
  starRocksFeSpec:
    config: |
      metastore_agent_max_connections = 32
      enable_iceberg_metadata_disk_cache = true
  starRocksCnSpec:
    config: |
      object_storage_max_connection = 1024
      datacache_enable = true
Step B: Use a Kubernetes Job for Global Variables Create a simple Job that runs once when you deploy or update your cluster.
Copy code
yaml
apiVersion: batch/v1
kind: Job
metadata:
  name: sr-init-config
spec:
  template:
    spec:
      containers:
      - name: mysql-client
        image: mysql:8.0
        command: ["/bin/sh", "-c"]
        args:
          - |
            # Wait for FE to be ready
            until mysql -h starrocks-fe-service -P9030 -uroot -e "SELECT 1"; do
              echo "Waiting for FE..."
              sleep 5
            done
            # Apply idempotent global settings
            mysql -h starrocks-fe-service -P9030 -uroot <<EOF
            SET GLOBAL plan_mode = 'distributed';
            SET GLOBAL enable_iceberg_column_statistics = true;
            SET GLOBAL connector_io_tasks_per_scan_operator = 8;
            EOF
      restartPolicy: OnFailure
4. Summary of Variable Scopes Variable | Category | How to Persist? 
plan_mode
| Session/Global | 
SET GLOBAL
(Persisted in Metadata PV) 
enable_iceberg_column_statistics
| Session/Global | 
SET GLOBAL
(Persisted in Metadata PV) 
connector_io_tasks_per_scan_operator
| Session/Global | 
SET GLOBAL
(Persisted in Metadata PV) 
metastore_agent_max_connections
| FE Config | 
fe.conf
(via Operator CR) 
object_storage_request_timeout_ms
| CN Config | 
cn.conf
(via Operator CR) Safety Concerns & Racing
* Race Conditions: If you use a 
postStart
hook, it executes the moment the container starts. Since the FE needs time to load metadata and elect a leader, the hook will likely fail. The Job pattern with a 
until
loop is the safest way to handle the FE's internal readiness. * Idempotency: 
SET GLOBAL
is natively idempotent. Running it multiple times has no side effects, making it safe for GitOps/ArgoCD pipelines. Rocky's Final Advice: Treat your cluster as Immutable Infrastructure. While 
SET GLOBAL
survives pod restarts via the PV, always keep a copy of your initialization SQL in a ConfigMap or Job so you can recreate the entire environment from scratch in a year from now without losing your performance optimizations! References • sql-reference/System_variable.md • administration/management/FE_configuration.md • administration/management/BE_configuration.md
Open in Slack
PreviousNext
Powered by