http://starrocks.io logo
Join Slack
Powered by
Hi Team and <@U0A71G31CDV>, I want to enable Data ...
# questions-and-troubleshooting
r
Hi Team and @Rocky, I want to enable Data Caching in my CN pods. For that I did below configurations in starrcoks helm chart values:
Copy code
config: |
    sys_log_level = INFO
    # ports for admin, web, heartbeat service
    be_port = 9060
    webserver_port = 8040
    heartbeat_service_port = 9050
    brpc_port = 8060
    datacache_disk_size = 80%
    storage_root_path = /opt/starrocks/cn/storage0;/opt/starrocks/cn/storage1
and
Copy code
storageSpec:
  name: "cn-cache-storage"
  storageClassName: ""
  storageSize: 20Gi
  storageCount: 2
  storageMountPath: "/opt/starrocks/cn/storage"
  logStorageClassName: ""
  logStorageSize: 20Gi
  logMountPath: "/opt/starrocks/cn/log"
  spillStorageClassName: ""
  spillStorageSize: 0Gi
  spillMountPath: ""
But still when I check status of ComputeNodes, DiskUsage is 0B/0B
Copy code
Starrocks > show compute nodes\G;
*************************** 1. row ***************************
        ComputeNodeId: 10005
                   IP: kube-starrocks-cn-0.kube-starrocks-cn-search.ASDF.svc.cluster.local
        HeartbeatPort: 9050
               BePort: 9060
             HttpPort: 8040
             BrpcPort: 8060
        LastStartTime: 2026-03-12 20:52:40
        LastHeartbeat: 2026-03-12 21:02:35
                Alive: true
 SystemDecommissioned: false
ClusterDecommissioned: false
               ErrMsg:
              Version: 4.0.1-cd9df36
             CpuCores: 1
             MemLimit: 1.620GB
    NumRunningQueries: 0
           MemUsedPct: 12.47 %
           CpuUsedPct: 0.0 %
     DataCacheMetrics: Status: Normal, DiskUsage: 0B/0B, MemUsage: 18.9KB/331.7MB
       HasStoragePath: true
           StatusCode: OK
          StarletPort: 9070
             WorkerId: 1
        WarehouseName: default_warehouse
            TabletNum: 113
1 row in set (0.01 sec)
PVC got bounded successfully. Also seeing these warnings in the CN Pod logs:
Copy code
W20260312 20:52:37.895801 140066831936896 memory_lock.cpp:46] mlock failed for  143073280-349753344 (206680064 bytes): Cannot allocate memory
W20260312 20:52:38.018388 140066831936896 timezone_utils.cpp:93] not found timezone:Africa/Casablanca
W20260312 20:52:38.019061 140066831936896 timezone_utils.cpp:93] not found timezone:Africa/El_Aaiun
W20260312 20:52:38.110549 140066831936896 disk_space_monitor.cpp:192] The current available disk space is too small, so disable the disk cache directly. If you still need it, you could reduce the value of `datacache_min_disk_quota_for_adjustment`, path: /opt/starrocks/cn/storage1/datacache
W20260312 20:52:38.110610 140066831936896 disk_space_monitor.cpp:192] The current available disk space is too small, so disable the disk cache directly. If you still need it, you could reduce the value of `datacache_min_disk_quota_for_adjustment`, path: /opt/starrocks/cn/storage0/datacache
W20260312 20:52:38.667730 140066831936896 jit_engine.cpp:476] System or Process memory limit is less than 16GB, disable JIT. You can set jit_lru_cache_size or jit_lru_object_cache_size a properly positive value in BE's config to force enabling JIT
W20260312 20:52:40.293134 140063534380608 heartbeat_server.cpp:222] kube-starrocks-cn-0.kube-starrocks-cn-search.ASDF.svc.cluster.local not equal to to backend localhost 10.00.00.04
W20260312 20:55:22.648996 140065125992000 cache_file.h:130] write star cache file meta/0000000000000000_000000000000004F.meta failed, errmsg: INTERNAL: fail to write block because no valid cache space
W20260312 20:57:32.561349 140064717751872 cache_file.h:130] write star cache file data/00000000000003f1_e6fe1c0c-b84b-4457-b226-a5f771b1d78f.dat failed, errmsg: INTERNAL: fail to write block because no valid cache space
W20260312 21:06:34.014033 140064726144576 cache_file.h:130] write star cache file data/000000000000041c_f8885f53-5cf1-4ec0-a30f-721fcde280d2.dat failed, errmsg: INTERNAL: fail to write block because no valid cache space
r
Hello! It looks like you're encountering an issue where the Data Cache is being automatically disabled because StarRocks deems the available disk space too small for safe operation. Based on the logs and your configuration, here is what is happening and how to fix it: 1. Why it's happening Starting from StarRocks v3.3+, Data Cache Automatic Scaling is enabled by default. In v4.0.1, the system checks if the calculated disk quota for the cache meets a minimum threshold defined by 
datacache_min_disk_quota_for_adjustment
(default is 10 GB). Because your disks are 20 GiB each: * The system calculates a "safe" quota (default 
disk_safe_level
is 80%, which is ~16 GB). * However, if there's any existing usage on the disk or if the internal safety margins (like 
disk_high_level
) result in a calculated quota that fluctuates near the minimum, the system may decide to disable the cache entirely to avoid performance degradation from frequent evictions. * The log 
The current available disk space is too small, so disable the disk cache directly
confirms this safety mechanism triggered. 2. Recommended Fixes A. Disable Automatic Scaling and set a manual quota The most reliable way to force the cache to stay enabled on smaller disks is to disable the auto-adjustment feature and set the size manually. Add these to your 
config
block in `values.yaml`:
Copy code
yaml
config: |
    # Disable auto-scaling (Note: parameter name changed in v4.0)
    enable_datacache_disk_auto_adjust = false
    # Manually set the disk size (percentage or absolute value)
    datacache_disk_size = 15G
    # Explicitly enable datacache (good practice)
    datacache_enable = true
B. Lower the Minimum Quota Threshold If you want to keep automatic scaling enabled but allow it to work on your 20 GiB disks, you should lower the minimum adjustment threshold:
Copy code
yaml
config: |
    # Lower threshold from 10GB (default) to 2GB
    datacache_min_disk_quota_for_adjustment = 2147483648
3. Other Observations * Memory Limit: Your CN Pod has a very low memory limit (
1.620GB
). StarRocks logs show 
System or Process memory limit is less than 16GB, disable JIT
. While this doesn't stop the cache, it will significantly impact query performance. For production, 16GB+ is recommended. * mlock Warning: The 
mlock failed
warning is common in Kubernetes environments where the Pod doesn't have 
IPC_LOCK
capabilities or the memory limit is too tight. It's usually non-fatal but indicates the system can't lock memory for certain operations. * Path Alignment: Your 
storage_root_path
(
/opt/starrocks/cn/storage0
) matches the default mounting behavior of the operator when 
storageCount
is 2, so that part is correct. To apply changes: Update your 
values.yaml
with the parameters in 2A and redeploy the Helm chart. After the CN pods restart, run 
SHOW COMPUTE NODES\G
again to verify 
DiskUsage
is now reflecting your quota. Let me know if you need any more help! References • administration/management/BE_configuration.md • data_source/data_cache.md • data_source/data_cache_troubleshooting.md
r
@Rocky Yes this worked. I attached 2 PVC with size 20GiB each but than why diskUsage is showing 31.8GB?
Copy code
DataCacheMetrics: Status: Normal, DiskUsage: 20MB/31.8GB, MemUsage: 34.6MB/331.7MB
r
The reason you are seeing 31.8GB instead of the full 40GiB (2 x 20GiB) is due to the default configuration of the StarRocks Data Cache (StarCache). By default, StarRocks limits the Data Cache to use 80% of the available disk capacity on the configured paths. This is controlled by the parameter 
starlet_star_cache_disk_size_percent
. The Calculation: * Total Raw Capacity: 2 * 20 GiB = 40 GiB. * Default Limit (80%): 40 GiB * 0.8 = 32 GiB. * The small difference between 32 GiB and the 31.8 GB you see is typically due to filesystem overhead (reserved blocks) or differences in how units (GB vs GiB) are reported by the operating system. How to change this: If you want to allow the Data Cache to use more of your PVC space, you can modify the following BE (or CN) configuration parameter: * 
starlet_star_cache_disk_size_percent
: Set this to a higher value (e.g., 
90
) if you want to utilize more of the disk. * 
datacache_disk_size
: Alternatively, you can specify a fixed size in bytes (e.g., 
38654705664
for 36GiB). Note: It is generally recommended to keep some headroom (like the default 20%) to account for system logs, temporary files, and to prevent the disk from reaching 100% fullness, which can cause stability issues for the underlying filesystem. References • administration/management/BE_configuration.md • using_starrocks/caching/block_cache.md • deployment/shared_data/feature-support-shared-data.md
🙌 1
r
@Rocky After setting this in helm chart of starrocks,
Copy code
feEnvVars:
    - name: LOG_CONSOLE
      value: "1"
the logs are not getting appended in these files:
Copy code
drwxrwsr-x. 3 root      starrocks      223 Mar 13 11:29 .
drwxr-xr-x. 1 starrocks starrocks       71 Mar 13 11:29 ..
-rw-r--r--. 1 starrocks starrocks        0 Mar 13 11:29 fe.audit.log
-rw-r--r--. 1 starrocks starrocks        0 Mar 13 11:29 fe.big_query.log
-rw-r--r--. 1 starrocks starrocks        0 Mar 13 11:29 fe.dump.log
-rw-r--r--. 1 starrocks starrocks        0 Mar 13 11:29 fe.features.log
-rw-r--r--. 1 starrocks starrocks 20279024 Mar 16 18:31 fe.gc.log.20260313-112919
-rw-r--r--. 1 starrocks starrocks        0 Mar 13 11:29 fe.internal.log
-rw-r--r--. 1 starrocks starrocks        0 Mar 13 11:29 fe.log
-rw-r--r--. 1 starrocks starrocks        0 Mar 13 11:29 fe.profile.log
-rw-r--r--. 1 starrocks starrocks        0 Mar 13 11:29 fe.warn.log
The logs started coming in the stdout/stderr but not seeing audit logs.
r
r
@Rocky How to enable audit logs?
r
To enable and manage audit logs in StarRocks, you can either use the default file-based logging or install the AuditLoader plugin to store logs in a StarRocks table for SQL-based analysis. 1. Default File-based Audit Logs StarRocks enables audit logging by default. The logs are stored in the FE (Frontend) log directory. * Location: 
fe/log/fe.audit.log
* Default Modules: By default, it records 
query
and 
slow_query
. Configuration (fe.conf) You can modify the following parameters in your 
fe.conf
file to customize the behavior: * 
audit_log_modules
: Specifies which modules to log (e.g., 
query, slow_query, connection
). * 
audit_log_dir
: Changes the directory where audit logs are stored. * 
audit_log_roll_num
: Number of history files to retain (default: 90). * 
audit_log_delete_age
: How long to keep logs before deletion (default: 7 days). * 
audit_log_json_format
: Set to 
true
if you prefer logs in JSON format for easier parsing. ────────── 2. Managing Audit Logs via AuditLoader (Recommended) If you want to query audit logs using SQL statements, you can use the AuditLoader plugin. This plugin reads the local 
fe.audit.log
and loads it into a StarRocks table via Stream Load. Step 1: Create the target Database and Table Create a table in StarRocks to store the logs. The schema should match the fields in the audit log (which may vary slightly by version).
Copy code
sql
CREATE DATABASE starrocks_audit_db__;
-- Create table (Refer to official documentation for the specific CREATE TABLE schema 
-- for your version as fields like cpuCostNs or memCostBytes are added over time)
Step 2: Download and Configure the Plugin 1. Download the AuditLoader package from the StarRocks resources page. 2. Unzip and modify `plugin.conf`: * `frontend_host_port`: Set to 
127.0.0.1:8030
. * `database`: 
starrocks_audit_db__
. * `table`: The name of the table you created. * 
user
and `password`: Your StarRocks credentials (must have 
LOAD_PRIV
). 3. Zip the files back into 
auditloader.zip
. Step 3: Install the Plugin Dispatch the zip file to all FE nodes and run the following command in your MySQL client:
Copy code
sql
INSTALL PLUGIN FROM "/absolute/path/to/auditloader.zip";
Step 4: Verify Check the status of the plugin:
Copy code
sql
SHOW PLUGINS;
Once the 
Status
is 
INSTALLED
, you can query your audit logs after a short delay (usually 60 seconds):
Copy code
sql
SELECT * FROM starrocks_audit_db__.your_audit_table;
3. Summary of Key Parameters Parameter | Description | Default 
audit_log_modules
| Modules to log (query, slow_query, connection) | 
query, slow_query
qe_slow_log_ms
| Threshold for "slow" queries in milliseconds | 
5000
audit_log_json_format
| Enable JSON formatting | 
false
References • administration/management/FE_configuration.md • administration/management/logs.md • administration/management/audit_loader.md
r
@Rocky What are these logs means in fe pods?
Copy code
(starrocks-mysql-nio I/O-3|124) [AcceptListener.handleEvent():80] Connection established. remote=/10.218.100.21:51422, connectionId=33580986
r
These logs indicate that a new client connection has been successfully established with the StarRocks FE (Front End) via its MySQL-compatible interface. Here is a breakdown of what each part of the log means: * 
starrocks-mysql-nio I/O-3
: This is the name of the thread handling the event. It belongs to the NIO (Non-blocking I/O) thread pool that StarRocks uses to manage network communication for MySQL clients. * 
AcceptListener.handleEvent():80
: This refers to the specific code location in the FE source code where the "accept" event (a new incoming connection request) was processed. * 
Connection established.
: This confirms that the TCP handshake was successful and the FE has accepted the connection. * 
remote=/10.218.100.21:51422
: This is the source of the connection. * 
10.218.100.21
is the IP address of the client (could be a BI tool, an application, or a load balancer). * 
51422
is the ephemeral port used by the client. * 
connectionId=33580986
: This is a unique internal ID assigned by the FE to this specific session. You can often use this ID to correlate the connection with specific queries in the 
fe.audit.log
. Is this an error? No, this is a standard 
INFO
level log. It is normal behavior whenever a client connects to StarRocks. When should you be concerned? If you see an extremely high frequency of these logs (thousands per minute), it might indicate: 1. Short-lived connections: Your application is not using a connection pool, creating a new connection for every single query. 2. Health check noise: A load balancer or Kubernetes liveness/readiness probe is hitting the MySQL port frequently. 3. Connection Limit: If you reach the limit defined by the FE configuration 
qe_max_connection
(default is 1024 or 4096 depending on the version), new connections will be rejected. References • faq/operation_maintenance_faq.md • administration/management/logs.md • deployment/deploy_manually.md • administration/management/FE_configuration.md • administration/management/monitoring/metrics.md
r
@Rocky How to check 
qe_max_connection
?
r
To check the value of the 
qe_max_connection
parameter in StarRocks, you can use the following SQL command:
Copy code
sql
ADMIN SHOW FRONTEND CONFIG LIKE 'qe_max_connection';
Explanation: * 
qe_max_connection
: This is an FE (Frontend) configuration parameter that defines the maximum number of connections allowed to be established by all users to a single FE node. * Default Value: Starting from v3.1.12 and v3.2.7, the default is 
4096
(previously 
1024
). * Is Mutable: No (This parameter cannot be modified dynamically; it requires a restart of the FE after changing it in the 
fe.conf
file). If you want to see all configuration parameters, you can simply run:
Copy code
sql
ADMIN SHOW FRONTEND CONFIG;
References • administration/management/FE_configuration.md • sql-reference/System_variable.md • administration/management/BE_configuration.md
r
@Rocky After adding this :
Copy code
feEnvVars / cnEnvVars: - name: LOG_CONSOLE value: "1"
Do we still need to mount logMountPath?
r
Setting 
LOG_CONSOLE: "1"
(which sets the 
sys_log_to_console
parameter to true) redirects the main system logs to the standard output (stdout), allowing you to view them via 
kubectl logs
or collect them using a log aggregator like Fluentd, Loki, or ELK. Whether you still need to mount 
logMountPath
depends on your specific requirements for log persistence and diagnostics: *1. When you can safely omit `logMountPath`:* If you have a centralized log management system (like ELK, Grafana Loki, or Datadog) that automatically scrapes stdout from your Kubernetes pods, you generally do not need a persistent volume for logs. Removing the mount saves on storage costs and simplifies your deployment. *2. When you should still keep `logMountPath`:* Even with 
LOG_CONSOLE: "1"
, there are several reasons why you might still want a persistent mount: * Audit Logs: By default, FE audit logs (
fe.audit.log
) might not be redirected to the console depending on your version and Log4j configuration. If these are critical for your compliance, you should keep the mount or verify they are appearing in stdout. * Data Load Error Logs: BE/CN nodes store detailed error samples for failed imports in the 
storage/error_log
directory. These are useful for debugging failed 
STREAM LOAD
or 
ROUTINE LOAD
jobs and are typically accessed via the BE's HTTP API. * Diagnostic Artifacts: StarRocks writes 
pprof
heap snapshots and CPU profiles to the log directory. If the pod crashes or restarts and you don't have a persistent volume, these files will be lost, making it harder to troubleshoot the cause of a crash. * JVM/GC Logs: Java Garbage Collection logs for the FE are often written to files and may not be captured by the standard console redirection. * Persistence across restarts: 
kubectl logs
only provides a limited buffer. If a pod enters a crash loop, logs written to the ephemeral container layer are lost when the pod restarts, whereas a PVC mount preserves them for post-mortem analysis. Summary Recommendation * For Development/Testing: You can skip 
logMountPath
and rely on 
LOG_CONSOLE: "1"
. * For Production: It is recommended to keep 
logMountPath
 mounted to a persistent volume (even if small) to ensure that audit logs, load error logs, and diagnostic artifacts (like pprof and core dumps) are preserved for troubleshooting and compliance. Note: The 
sys_log_to_console
feature was officially introduced/enhanced in version v3.2.0. Ensure your StarRocks version supports this environment variable if you plan to rely on it. References • administration/management/BE_configuration.md • administration/management/FE_configuration.md • administration/management/logs.md
r
@Rocky What is the production recommended setup of FE + CN in kubernetes if I am going to access FE from within pods within the Kubernetes. ?
r
For a production-ready setup of StarRocks FE (Frontend) and CN (Compute Node) within a Kubernetes environment, where access is primarily from other pods in the cluster, the following architecture and configurations are recommended: 1. High Availability (HA) Architecture * FE Nodes: Deploy at least 3 Follower FE nodes. This ensures that the cluster remains available even if the Leader FE node fails. The Raft protocol handles leader election and metadata consistency. * CN Nodes: Use Compute Nodes (CN) for elastic scaling. CNs are stateless, making them ideal for Kubernetes deployments. In a production environment, it is highly recommended to use Shared-Data mode (separate storage and compute) with an S3-compatible object store (like AWS S3, MinIO, or GCS) for persistence. 2. Enable FQDN Access (Crucial for Kubernetes) In Kubernetes, Pod IP addresses are ephemeral. To ensure stable communication between FE replicas and between FE and CN: * Host Type: You must start FE nodes with 
--host_type FQDN
. This ensures that StarRocks registers nodes in its metadata using their Fully Qualified Domain Names (e.g., 
starrocks-fe-0.starrocks-fe-search.svc.cluster.local
) rather than their temporary Pod IPs. * Operator/Helm: If using the StarRocks Operator (recommended), this is typically handled automatically. 3. Service Configuration for Internal Access To access the FE from other pods (e.g., an application pod running a MySQL client or a BI tool), use the Kubernetes Service: * Service Type: Use a 
ClusterIP
service for internal-only access. * Connection String: Use the internal DNS name of the FE service:
Copy code
bash
    mysql -h <cluster-name>-fe-service.<namespace>.svc.cluster.local -P 9030 -u root
* Load Balancing: The Kubernetes 
ClusterIP
service provides basic round-robin load balancing across your FE replicas (Followers/Observers). 4. Key FE Configurations for Kubernetes Add these parameters to your 
fe.conf
to optimize for the K8s environment: * 
dns_cache_ttl_seconds
: Set to a low value (e.g., 
60
or even 
10
if IPs change very frequently) to ensure FE can quickly resolve updated Pod IPs via the K8s DNS. * 
priority_networks
: While FQDN is preferred, setting 
priority_networks
helps the node pick the correct network interface if the pod has multiple interfaces. 5. Resource Management For a production environment, avoid "Best Effort" pods. Use Guaranteed Quality of Service (QoS) by setting requests and limits to the same values: * FE: 8 CPU / 16GB RAM minimum. * CN: 16 CPU / 64GB RAM minimum (depending on workload). * Disks: Use Persistent Volume Claims (PVCs) for FE metadata storage. CNs are stateless, but they benefit from a small local disk for caching (
storage_root_path
). 6. Summary Checklist Component | Recommendation Deployment Method | StarRocks Operator (Standard for Production) FE Count | 3 (Followers) CN Count | 3+ (Scalable based on query load) Access Mode | FQDN (
--host_type FQDN
) Internal Access | Use 
<service-name>.<namespace>.svc.cluster.local
Storage | Object Storage (Shared-Data mode) for best elasticity Note: If you are using "Shared-Nothing" (traditional BEs with local disks), use a 
StatefulSet
for BEs to ensure data persistence across pod restarts. However, for K8s, the Shared-Data architecture with CNs is the modern production recommendation for easier scaling and maintenance. References • deployment/plan_cluster.md • faq/Deploy_faq.md • quick_start/shared-data.md • deployment/deploy_manually.md • deployment/environment_configurations.md
r
@Rocky This is my current helm value for fe+cn setup. Can you identify any issue in this?
Copy code
starrocksCluster:
  namespace: "plattest02"
  componentValues:
    serviceAccount: qualkubvir-plattest02-starrocks-s3-sa
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
          - matchExpressions:
              - key: namespace
                operator: In
                values: ["plattest02"]
              - key: <http://karpenter.sh/capacity-type|karpenter.sh/capacity-type>
                operator: In
                values: [ "on-demand" ]
              - key: <http://kubernetes.io/arch|kubernetes.io/arch>
                operator: In
                values:
                  - amd64
              - key: <http://kubernetes.io/os|kubernetes.io/os>
                operator: In
                values:
                  - linux
# ============================================================
# FE-PROXY — EXPOSE THROUGH ALB (HTTP only)
# ============================================================
starrocksFeProxySpec:
  enabled: true
  image:
    tag: 1.28.0
  resources:
    requests:
      cpu: 1
      memory: 2Gi
    limits:
      cpu: 1
      memory: 2Gi
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
          - matchExpressions:
              - key: namespace
                operator: In
                values: ["plattest02"]
              - key: <http://karpenter.sh/capacity-type|karpenter.sh/capacity-type>
                operator: In
                values: [ "on-demand" ]
              - key: <http://kubernetes.io/arch|kubernetes.io/arch>
                operator: In
                values:
                  - amd64
              - key: <http://kubernetes.io/os|kubernetes.io/os>
                operator: In
                values:
                  - linux
# ============================================================
# FE (FrontEnd Service) — EXPOSE THROUGH NLB (MySQL + HTTP)
# ============================================================
starrocksFESpec:
  image:
    tag: "4.0.1"
  resources:
    requests:
      cpu: 3
      memory: 8Gi
    limits:
      cpu: 3
      memory: 8Gi
  config: |
    run_mode = shared_data
    cloud_native_storage_type = S3
    aws_s3_path = qualkubvir-plattest02-starrocks-data
    aws_s3_region = us-east-1
    aws_s3_endpoint = <https://s3.us-east-1.amazonaws.com>
    enable_load_volume_from_conf=true
    aws_s3_use_instance_profile=false
    aws_s3_use_aws_sdk_default_behavior=true
    enable_trace_historical_node = true
    audit_log_modules = slow_query, query, connection
    audit_log_json_format = true
    qe_slow_log_ms = 1000
    aws_s3_enable_partitioned_prefix=true
    aws_s3_num_partitioned_prefix=64
    automated_cluster_snapshot_interval_seconds=600
  storageSpec:
    storageSize: 10Gi
    logStorageSize: 10Gi
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
          - matchExpressions:
              - key: namespace
                operator: In
                values: [ "plattest02" ]
              - key: <http://karpenter.sh/capacity-type|karpenter.sh/capacity-type>
                operator: In
                values: [ "on-demand" ]
              - key: <http://kubernetes.io/arch|kubernetes.io/arch>
                operator: In
                values:
                  - amd64
              - key: <http://kubernetes.io/os|kubernetes.io/os>
                operator: In
                values:
                  - linux
# ============================================================
# CN (Compute Node)
# ============================================================
starrocksCnSpec:
  replicas: 1
  image:
    tag: "4.0.1"
  resources:
    requests:
      cpu: 8
      memory: 16Gi
    limits:
      cpu: 8
      memory: 16Gi
  storageSpec:
    storageSize: 20Gi
    logStorageSize: 20Gi
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
          - matchExpressions:
              - key: namespace
                operator: In
                values: [ "plattest02" ]
              - key: <http://kubernetes.io/arch|kubernetes.io/arch>
                operator: In
                values:
                  - amd64
              - key: <http://kubernetes.io/os|kubernetes.io/os>
                operator: In
                values:
                  - linux
  autoScalingPolicy: # Automatic scaling policy of the CN cluster.
    maxReplicas: 10 # The maximum number of CNs is set to 10.
    minReplicas: 1 # The minimum number of CNs is set to 1.
    # operator creates an HPA resource based on the following field.
    # see <https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/> for more information.
    hpaPolicy:
      metrics: # Resource metrics
        - type: Resource
          resource:
            name: memory  # The average memory usage of CNs is specified as a resource metric.
            target:
              # The elastic scaling threshold is 60%.
              # When the average memory utilization of CNs exceeds 60%, the number of CNs increases for scale-out.
              # When the average memory utilization of CNs is below 60%, the number of CNs decreases for scale-in.
              averageUtilization: 60
              type: Utilization
        - type: Resource
          resource:
            name: cpu # The average CPU utilization of CNs is specified as a resource metric.
            target:
              # The elastic scaling threshold is 60%.
              # When the average CPU utilization of CNs exceeds 60%, the number of CNs increases for scale-out.
              # When the average CPU utilization of CNs is below 60%, the number of CNs decreases for scale-in.
              averageUtilization: 60
              type: Utilization
      behavior: #  The scaling behavior is customized according to business scenarios, helping you achieve rapid or slow scaling or disable scaling.
        scaleUp:
          policies:
            - type: Pods
              value: 1
              periodSeconds: 10
        scaleDown:
          policies:
            - type: Pods
              value: 1
              periodSeconds: 30
          stabilizationWindowSeconds: 300
          selectPolicy: Max
r
Hello! I'm Rocky. I've reviewed your Helm configuration for the StarRocks cluster. There are several critical issues and potential improvements you should address to ensure the cluster deploys and functions correctly in shared-data mode. 1. Versioning Issue * StarRocks Version: You are using 
tag: "4.0.1"
. As of now, the latest stable major versions of StarRocks are in the 3.x series (e.g., 3.2, 3.3). Version 4.0.1 does not officially exist yet. * Recommendation: Use a stable version like 
3.2.11
or 
3.3.2
. 2. Missing CN Configuration for Shared-Data In a shared-data architecture, all nodes (both FE and CN) must be aware that they are running in shared-data mode. Your 
starrocksCnSpec
is missing the 
config
block. * Issue: Without 
run_mode = shared_data
, the CN will default to "shared-nothing" and will fail to join the FE. * Recommendation: Add the following to your `starrocksCnSpec`:
Copy code
yaml
    starrocksCnSpec:
      config: |
        run_mode = shared_data
3. Invalid Node Affinity Key * Issue: You have used 
- key: namespace
under 
nodeSelectorTerms
. * Problem: Kubernetes nodes are cluster-scoped resources and do not have a 
namespace
label by default. Unless you have manually labeled your nodes with 
namespace=plattest02
, your Pods will remain in a Pending state because no nodes will match this affinity. * Recommendation: Use a standard node label (like `topology.kubernetes.io/zone`) or a custom label you've applied to your nodes (e.g., 
workload-type: starrocks
). 4. Service Account for CN * Issue: You defined 
serviceAccount
under 
componentValues
, but for IRSA (IAM Roles for Service Accounts) to work in EKS, both FE and CN pods need the 
serviceAccountName
specified so they can access S3. * Recommendation: Explicitly add the 
serviceAccount
to both specs:
Copy code
yaml
    starrocksFESpec:
      serviceAccount: qualkubvir-plattest02-starrocks-s3-sa
    starrocksCnSpec:
      serviceAccount: qualkubvir-plattest02-starrocks-s3-sa
5. Shared-Data Cache Size (CN Storage) * Issue: Your CN 
storageSize
is set to 
20Gi
. * Problem: In shared-data mode, CNs use local storage as a Data Cache. If your hot data exceeds 20Gi, query performance will drop significantly as the CN will have to fetch data from S3 frequently. * Recommendation: Increase this to at least 100Gi-200Gi if your dataset is large, or ensure it's backed by fast SSDs (gp3). 6. HPA Strategy (Memory Target) * Issue: Scaling CNs based on 
averageUtilization: 60
for memory can be tricky. * Problem: StarRocks CNs are designed to use memory aggressively for caching and processing. If the cache fills up, memory usage will stay high, which might trigger your HPA to scale out to 
maxReplicas
even if the CPU load is low. * Recommendation: Consider increasing the memory threshold to 
80
or primarily relying on CPU utilization for scaling. 7. FE Configuration Tweaks * S3 Path: Ensure 
aws_s3_path
is just the bucket name or 
bucket/prefix
. * Instance Profile vs IRSA: Since you are using a 
serviceAccount
, having 
aws_s3_use_instance_profile=false
and 
aws_s3_use_aws_sdk_default_behavior=true
is correct for IRSA. Summary Checklist 1. Change image tags to a valid 3.x version. 2. Add 
run_mode = shared_data
to the 
starrocksCnSpec.config
. 3. Fix the 
nodeAffinity
key (remove 
namespace
unless manually labeled). 4. Ensure 
serviceAccount
is applied to both FE and CN specs. 5. (Optional) Increase CN memory to 32Gi if possible; StarRocks performs much better with more headroom. Would you like me to provide a corrected version of the full YAML? References • deployment/helm.md • deployment/shared_data/s3.md • deployment/shared_data/minio.md
r
@Rocky Which AWS Graviton instance do you recommend for FE+CN in EKS?
r
Based on StarRocks documentation and performance benchmarks, StarRocks has been significantly optimized for the ARM64 architecture, specifically AWS Graviton instances, starting from version 3.3. Recommended Graviton Instances For a production environment, we recommend the following AWS Graviton instance types based on the typical resource requirements of StarRocks components: 1. For BE (Backends) - Compute & Memory Intensive BE nodes perform the heavy lifting for data processing and storage. StarRocks recommends a minimum of 16 CPU cores and 64 GB RAM per BE node. * Recommended Instance: 
r7g
or 
r6g
series (e.g., 
r7g.4xlarge
). * Why: StarRocks is memory-intensive, especially for complex queries and high-concurrency scenarios. The 
r
series provides a higher memory-to-vCPU ratio (8 GiB per vCPU), which aligns well with the recommendation of 4-8 GB of RAM per CPU core. * Alternative: 
m7g
series (e.g., 
m7g.4xlarge
) if your memory requirements are lower (4 GiB per vCPU). 2. For FE (Frontends) - Management & Metadata FE nodes handle metadata management and query planning. They are generally less resource-intensive than BEs. StarRocks recommends 8 CPU cores and 16 GB RAM per FE node. * Recommended Instance: 
m7g.2xlarge
 or 
c7g.2xlarge
. * Why: These provide a balanced or compute-optimized configuration suitable for the FE's Java-based metadata operations. Key Performance Insights According to the StarRocks 3.3 Release Notes, AWS Graviton instances have shown superior performance compared to traditional x86 instances in several benchmarks: * SSB 100G: 11% faster than x86. * Clickbench: 39% faster than x86. * TPC-H 100G: 13% faster than x86. * TPC-DS 100G: 35% faster than x86. Deployment Tips for Graviton * Version: Ensure you are using StarRocks v3.3 or later to take advantage of the specific ARM architecture instruction set optimizations. * OS: Use a supported Linux distribution like Amazon Linux 2023, Ubuntu 22.04, or CentOS 7.9. * Storage: For the best performance, use EBS gp3 or io2 volumes, or instance-store (NVMe) if using 
is
variants (e.g., 
r7gd
) for data caching or temporary storage. For more details on cluster planning, you can refer to the Plan StarRocks cluster documentation. References • best_practices/audit_log_resource_group.md • faq/resource_isolation_faq.md • best_practices/partitioning.md
For running StarRocks FE and CN on AWS Graviton (ARM) within EKS, I recommend using the Graviton 3 (7g series) instances. Since StarRocks 3.3, the engine has been significantly optimized for ARM architecture, with benchmarks showing performance gains of 11% to 39% over x86 in various tests. Based on the standard StarRocks resource recommendations, here are the specific instance types for each component: 1. FE (Frontend) Nodes FE nodes primarily handle metadata, query planning, and connection management. They are less compute-intensive but require stability. * Recommended Instance: 
c7g.2xlarge
 (8 vCPUs, 16 GB RAM) * Alternative: 
m7g.2xlarge
 (8 vCPUs, 32 GB RAM) if you have a very large number of tables or complex metadata requirements. * Minimum Specs: 8 vCPUs and 16 GB RAM per node. For high availability, you should deploy at least 3 FE nodes. 2. CN (Compute) Nodes CN nodes handle the heavy lifting of query execution and data processing. These should be scaled according to your query complexity and data volume. * Recommended Instance: 
m7g.4xlarge
 (16 vCPUs, 64 GB RAM) * Scaling Up: For larger workloads, 
m7g.8xlarge
 (32 vCPUs, 128 GB RAM) is a standard high-performance choice. * Why M7g? StarRocks performs best with a roughly 1:4 vCPU-to-RAM ratio for compute-intensive workloads. The 
m7g
family matches this perfectly (4 GB per vCPU). If your queries involve extremely large joins that exceed memory, you might consider the 
r7g
 family (8 GB per vCPU). Key Recommendations for EKS: * StarRocks Version: Ensure you are using StarRocks v3.3 or later to take full advantage of the ARM-specific optimizations. * Docker Images: Make sure to pull the ARM64-specific tags of the StarRocks images (e.g., 
starrocks-fe-ubuntu:3.3.0
or higher, which are often multi-arch). * Node Groups: Create a dedicated Managed Node Group in EKS using the 
ARM_64
capacity type. * Avoid Mixed Architecture: It is recommended to keep your entire StarRocks cluster (all FEs and CNs) on the same architecture (ARM) to avoid performance heterogeneity issues. Note: While older documentation emphasized x86/AVX2, the latest versions (3.3+) have been specifically tuned for AWS Graviton's instruction sets, making them an excellent choice for price/performance in EKS. References • deployment/environment_configurations.md • deployment/plan_cluster.md • faq/resource_isolation_faq.md • best_practices/audit_log_resource_group.md • deployment/deployment_prerequisites.md
r
@Rocky Can FE and CN be deployed on Graviton Instances. What is your recommendation regarding this?
r
Yes, both FE (Front-end) and CN (Compute Node) can be deployed on AWS Graviton (ARM64) instances. In fact, StarRocks has significantly optimized its performance for the ARM architecture in recent versions. Compatibility * FE (Front-end): Since the FE is Java-based, it runs on any architecture that supports a compatible JVM (JDK 11+ for v3.3/3.4, JDK 17+ for v3.5+). * CN/BE (Compute Node/Backend): These are C++ based. While earlier versions of StarRocks were primarily optimized for x86 (AVX2), StarRocks has introduced specialized SIMD optimizations for ARM (using NEON/SVE instructions). Performance & Recommendations Based on the latest StarRocks documentation (specifically starting from v3.3): 1. High Performance on Graviton: Performance tests on AWS Graviton instances with StarRocks 3.3+ showed that the ARM architecture can actually outperform x86 in several benchmarks (e.g., ~11% faster in SSB 100G and up to 39% faster in Clickbench). 2. Recommended Instance Types: * For CN/BE: We recommend using Graviton 3 or Graviton 4 instances (such as c7g, r7g, or the newer c8g/r8g series). These provide the best price-performance ratio. * Instance Sizing: Consistent with x86, we recommend at least 16 vCPUs and 64 GB RAM per CN/BE node for production workloads to handle complex vectorized queries effectively. 3. Mixing Architectures: It is not recommended to mix x86 and ARM nodes within the same cluster (e.g., having some x86 BEs and some Graviton BEs). Hardware heterogeneity can lead to unpredictable performance and potential issues during query execution. 4. Version Selection: If you plan to use Graviton, ensure you are using StarRocks v3.3 or later to take advantage of the specific ARM instruction set optimizations. Summary Table Component | Supported on Graviton? | Recommendation FE | Yes | Any Graviton instance with 8 vCPUs / 16GB+ RAM. CN / BE | Yes | c7g or r7g (Graviton 3) or newer. Use v3.3+ for best performance. When deploying, make sure to download the specific ARM64/AARCH64 binary packages or use the corresponding Docker images. References • deployment/deployment_prerequisites.md • faq/Deploy_faq.md • deployment/plan_cluster.md • faq/Others.md • introduction/Architecture.md • release_notes/release-3.3.md • release_notes/release-2.4.md
r
@Rocky I am getting this error while creating databse from a new user which has permission to create database:
Copy code
{
  "exception": "io.vertx.mysqlclient.MySQLException: {errorMessage=Access denied; you need (at least one of) the CREATE DATABASE privilege(s) on CATALOG default_catalog for this operation. Please ask the admin to grant permission(s) or try activating existing roles using <set [default] role>. Current role(s): NONE. Inactivated role(s): [sr_admin_role]., errorCode=5203, sqlState=42000}\n"
}
r
The error you're seeing occurs because, although the user has been granted the 
sr_admin_role
, that role is not active in your current session. In StarRocks, roles must be activated to use the privileges associated with them. You can resolve this using one of the following methods: 1. Activate the role for the current session If you just want to fix this for your current connection, run the following SQL command:
Copy code
sql
SET ROLE sr_admin_role;
-- Or to activate all roles assigned to you:
SET ROLE ALL;
After running this, you should be able to create the database. 2. Set the role as a Default Role (Recommended) To ensure this role is automatically activated every time the user logs in, you can set it as the default role. If you are logged in as the new user:
Copy code
sql
SET DEFAULT ROLE sr_admin_role;
If you are an admin (e.g., 
root
) setting it for that user:
Copy code
sql
ALTER USER 'username'@'%' DEFAULT ROLE sr_admin_role;
3. Automatically activate all roles for all users (Global) If you want StarRocks to automatically activate every granted role for every user upon login (similar to how some other databases behave), an admin can run:
Copy code
sql
SET GLOBAL activate_all_roles_on_login = TRUE;
Note: This applies to all users in the cluster going forward. How to verify You can check which roles are currently active in your session by running:
Copy code
sql
SELECT current_role();
If it returns 
NONE
, you will continue to get "Access denied" errors for operations requiring specific role privileges. References • administration/user_privs/authorization/privilege_faq.md • administration/user_privs/authorization/User_privilege.md • administration/user_privs/authorization/user_privs.md
r
@Rocky How to SET enable_profile = true; In helm chart of starrcoks fe config?
r
Setting 
enable_profile = true
is typically done as a system variable (session or global level) via SQL, rather than a direct parameter in the 
fe.conf
file. However, if you want to ensure query profiles are collected by default in a StarRocks cluster deployed via Helm, you have two primary options: 1. Set the FE Configuration (fe.conf) via Helm In the FE configuration, there is a parameter called 
enable_collect_query_detail_info
that controls whether the system collects query profiles. You can set this in your 
values.yaml
file for the Helm chart (assuming you are using the 
kube-starrocks
chart or StarRocks Operator):
Copy code
yaml
starrocksFESpec:
  config: |
    enable_collect_query_detail_info = true
* Note: This tells the FE to collect profiles. However, it is slightly different from the session variable 
enable_profile
, which specifically instructs the BE to send the profile to the FE for analysis. 2. Set the Variable Globally via SQL (Recommended) Since 
enable_profile
is a system variable, the standard way to enable it for all future sessions is to run a 
SET GLOBAL
command after the cluster is up. You can do this once using any MySQL client:
Copy code
sql
SET GLOBAL enable_profile = true;
This is the most reliable way to ensure that every query's profile is sent for analysis without needing to restart the FE. Summary of Differences * 
enable_profile
(System Variable): Controls if the BE sends the profile to the FE. Default is 
false
. * 
enable_collect_query_detail_info
(FE Config): Controls if the FE collects and stores the profile details (available in FE Web UI). If you are looking to troubleshoot performance and want to see profiles in the FE Web UI (
http://<FE_IP>:<FE_HTTP_PORT>/query
), you should ideally set both: 1. FE Config: 
enable_collect_query_detail_info = true
(in Helm 
values.yaml
). 2. Global Variable: 
SET GLOBAL enable_profile = true;
(via SQL). References • administration/management/FE_configuration.md • sql-reference/System_variable.md • faq/Others.md
r
@Rocky What is statistics database used for and what is the TTL of records saved in it?
r
The 
_statistics_
database in StarRocks is an internal system database primarily used to store metadata and statistical information required by the Cost-Based Optimizer (CBO) to generate efficient query execution plans. Purpose of 
_statistics_
Database It contains several tables that store different types of statistics gathered from your data: * 
column_statistics
: Stores basic statistics for tables and columns, such as row counts, number of distinct values (NDV), null counts, and min/max values. * 
histogram_statistics
: Stores histogram data used to handle data skew and provide more accurate selectivity estimation for non-uniform data. * 
external_column_statistics
: Stores statistics for data in external catalogs (like Hive, Iceberg, etc.). * 
analyze_job
& 
analyze_status
: Track the configuration and execution history of statistics collection tasks (jobs). * 
multi_column_statistics
: (Available since v3.5) Stores joint statistics for multiple columns to improve estimation for correlated predicates. ────────── TTL and Data Retention The retention of records in the 
_statistics_
database is governed by several FE (Frontend) configuration parameters: 1. Collection Task History (
analyze_status
): * Parameter: 
statistic_analyze_status_keep_second
* Default: 
259200
(3 days) * Description: This determines how long the history of statistics collection tasks is kept in the 
analyze_status
table. 2. Actual Statistics (
column_statistics
, 
histogram_statistics
): * These records do not have a fixed time-based TTL. Instead, they are kept as long as the corresponding table or partition exists. * Expiration/Cleanup: When a table or partition is dropped, StarRocks identifies the associated statistics as "expired." The system cleans these up based on the interval defined by: * Parameter: 
statistic_manager_sleep_time_sec
* Default: 
60
(1 minute) * Update Frequency: Records are overwritten whenever a new 
ANALYZE
task (manual or automatic) completes for that specific table/column. 3. Other Internal History (e.g., 
loads_history
): * Note that while 
_statistics_
is for CBO, there is also a 
statistics
database (no underscores) containing system views like 
loads_history
. * 
loads_history
: Stores load records for the last 3 months by default (this can be adjusted by modifying the 
partition_ttl
of the underlying table). How to Check/Modify You can check these settings using:
Copy code
sql
ADMIN SHOW FRONTEND CONFIG LIKE '%statistic%';
To modify a TTL (e.g., keeping task history for 7 days):
Copy code
sql
ADMIN SET FRONTEND CONFIG ("statistic_analyze_status_keep_second" = "604800");
References • using_starrocks/Cost_based_optimizer.md • knowledge/admin_and_operation/shared_data_compaction.md
r
@Rocky Why the date in starrocks pods coming in EDT format and not in UTC. How to bring it in UTC format. I have added this in my starrocks helm chart values
Copy code
timeZone: America/New_York
And date command inside the pod is coming like this:
Copy code
starrocks@kube-starrocks-fe-0:/opt/starrocks$ date
Thu Mar 26 08:51:53 EDT 2026
r
The reason your StarRocks pods are showing the date in EDT is because of the configuration you've applied in your Helm chart:
Copy code
yaml
timeZone: America/New_York
When you set this value in the StarRocks Helm chart (or the StarRocks Operator), it configures the container's underlying operating system timezone (usually by mounting 
/etc/localtime
or setting the 
TZ
environment variable). Since 
America/New_York
currently observes Daylight Saving Time (EDT), the 
date
command and the StarRocks 
system_time_zone
variable will reflect that. How to bring it to UTC format To change the pod environment and the database's reference time to UTC, follow these steps: 1. Update your Helm Chart values Change the 
timeZone
parameter in your 
values.yaml
to 
UTC
(or 
Etc/UTC
):
Copy code
yaml
# Change this:
# timeZone: America/New_York

# To this:
timeZone: "UTC"
2. Apply the changes Update your Helm release to roll out the change to the pods:
Copy code
bash
helm upgrade <release_name> <chart_path> -f values.yaml
3. Verify the changes Once the pods have restarted, check the date inside the pod:
Copy code
bash
kubectl exec -it <fe-pod-name> -- date
# Expected output: Thu Mar 26 12:51:53 UTC 2026
4. Update StarRocks Global Timezone (Optional but Recommended) Even if the system timezone is UTC, StarRocks has an internal 
time_zone
variable that defaults to 
Asia/Shanghai
in some versions or follows the system. To ensure the database functions (like 
now()
) return UTC, run the following SQL command:
Copy code
sql
-- Set the global timezone to UTC
SET GLOBAL time_zone = 'UTC';
Why this matters * system_time_zone: This is a read-only variable in StarRocks that is determined by the pod's environment at startup. * time_zone: This is the session/global variable that determines how 
DATETIME
values are displayed and calculated. It is best practice to keep 
time_zone
aligned with your 
system_time_zone
to avoid confusion during data loading. References • deployment/environment_configurations.md • administration/management/timezone.md • integrations/loading_tools/DataX-starrocks-writer.md • sql-reference/sql-statements/loading_unloading/BROKER_LOAD.md • sql-reference/sql-statements/loading_unloading/routine_load/CREATE_ROUTINE_LOAD.md • loading/Spark-connector-starrocks.md • sql-reference/System_variable.md • faq/loading/Broker_load_faq.md
r
@Rocky How to check what is the current timezone in starrocks ?
Copy code
SET GLOBAL time_zone = 'UTC';
r
To check the current timezone in StarRocks, you can use any of the following methods: 1. Using 
SHOW VARIABLES
 This is the most common way to see both the session-level timezone and the system-wide timezone:
Copy code
sql
SHOW VARIABLES LIKE '%time_zone%';
* 
time_zone
: Shows the timezone for your current session. * 
system_time_zone
: Shows the timezone of the server where the FE (Front End) is running. 2. Using the 
current_timezone()
function You can also call this function directly in a query:
Copy code
sql
SELECT current_timezone();
3. Checking with 
SELECT NOW()
 Since the 
NOW()
function depends on the current timezone setting, you can verify it by checking the current time:
Copy code
sql
SELECT NOW();
Quick Summary of Setting Timezones: * Session Level (only affects your current connection):
Copy code
sql
    SET time_zone = 'UTC';
* Global Level (affects all new connections):
Copy code
sql
    SET GLOBAL time_zone = 'UTC';
Note: The value can be a name (e.g., 
'Asia/Shanghai'
, 
'America/Los_Angeles'
) or an offset (e.g., 
'+08:00'
). StarRocks defaults to 
Asia/Shanghai
if not configured otherwise. References • administration/management/timezone.md • faq/Sql_faq.md • faq/loading/Broker_load_faq.md • loading/Spark-connector-starrocks.md • sql-reference/sql-functions/date-time-functions/current_timezone.md • sql-reference/System_variable.md • deployment/environment_configurations.md
r
@Rocky I got this:
Copy code
SHOW VARIABLES LIKE '%time_zone%';
+------------------+---------------+
| Variable_name    | Value         |
+------------------+---------------+
| system_time_zone | UTC           |
| time_zone        | Asia/Shanghai |
+------------------+---------------+
2 rows in set (0.01 sec)
I want to set timezone as in UTC
r
To set the 
time_zone
to UTC, you can do so at either the session level or the global level. 1. Set for the current session If you only want to change the timezone for your current connection:
Copy code
sql
SET time_zone = 'UTC';
Note: This setting will be lost once you disconnect. 2. Set globally If you want this to apply to all future sessions and persist across FE restarts:
Copy code
sql
SET GLOBAL time_zone = 'UTC';
Note: After running this, existing sessions will keep their old timezone. Only new connections will use UTC. 3. Using UTC Offset You can also use the offset format if preferred:
Copy code
sql
SET time_zone = '+00:00';
Why this is recommended As your 
system_time_zone
is already 
UTC
, it is highly recommended to set your 
time_zone
to 
UTC
as well. According to StarRocks documentation, having these two values match helps ensure that 
DATE
and 
DATETIME
types are handled correctly during data loading. You can verify the change again with:
Copy code
sql
SHOW VARIABLES LIKE '%time_zone%';
References • administration/management/timezone.md • faq/loading/Broker_load_faq.md • loading/Spark-connector-starrocks.md • integrations/loading_tools/DataX-starrocks-writer.md • sql-reference/System_variable.md • faq/Sql_faq.md • sql-reference/sql-statements/loading_unloading/BROKER_LOAD.md
r
@Rocky How toi set time_zone to UTC during the cluster creation via helm chart?
r
To set the time zone to UTC during cluster creation via the StarRocks Helm chart (using the StarRocks Operator), you should follow two main steps: configuring the container environment and setting the global system variable. 1. Set the Pod Environment Variable (
TZ
) You can configure the operating system time zone of the FE and BE pods by adding the 
TZ
environment variable in your 
values.yaml
file. This ensures that the 
system_time_zone
in StarRocks is identified as UTC. In your `values.yaml`:
Copy code
yaml
starrocksFESpec:
  # ... other FE configs
  envVars:
    - name: TZ
      value: "UTC"

starrocksBeSpec:
  # ... other BE configs
  envVars:
    - name: TZ
      value: "UTC"
2. Configure the Global Time Zone Variable While the 
TZ
environment variable sets the system time, StarRocks' default global 
time_zone
variable is 
Asia/Shanghai
. Currently, this cannot be set directly in 
fe.conf
. You must execute a SQL command after the cluster is created to change the default for all future sessions. Once the cluster is running, connect via a MySQL client and run:
Copy code
sql
SET GLOBAL time_zone = 'UTC';
Why both? * 
system_time_zone
: Is an immutable variable that StarRocks inherits from the machine (or Pod) OS environment. Setting 
TZ=UTC
in the Helm chart ensures this matches your expectation. * 
time_zone
: Is the session/global variable used for time calculations (like 
NOW()
or 
FROM_UNIXTIME
). Setting this globally ensures consistency across all user sessions. Summary of 
values.yaml
Example If you are using the 
kube-starrocks
chart, your configuration might look like this:
Copy code
yaml
starrocks:
  starrocksFESpec:
    replicas: 3
    envVars:
      - name: TZ
        value: "UTC"
  starrocksBeSpec:
    replicas: 3
    envVars:
      - name: TZ
        value: "UTC"
Note: After deployment, you can verify the settings by running:
Copy code
sql
SHOW VARIABLES LIKE '%time_zone%';
References • administration/management/timezone.md • deployment/environment_configurations.md • faq/loading/Broker_load_faq.md • quick_start/helm.md • integrations/loading_tools/DataX-starrocks-writer.md • loading/Spark-connector-starrocks.md
r
@Rocky What is the recommended CPU and Memory needs to be allocated for FE and CN pods, And based on what paramters it is decided?
r
Recommended resource allocation for StarRocks FE and CN pods in a production environment is based on the following guidelines: 1. FE (Front End) Pods The FE is responsible for metadata management, query planning, and scheduling. * Recommended CPU: 8 Cores. * Recommended Memory: 16 GB (Minimum). Parameters for Decision: * Number of Tablets: This is the most critical factor for FE memory. StarRocks stores metadata in memory. * < 1 Million Tablets: 16 GB RAM. * 1 – 2 Million Tablets: 32 GB RAM. * 2 – 5 Million Tablets: 64 GB RAM. * 5 – 10 Million Tablets: 128 GB RAM. * Query Concurrency: If you have a very high volume of concurrent small queries, you may need more CPU for the FE to handle connection overhead and query parsing. In such cases, adding Observer FE nodes is often better than just increasing resources on existing ones. * JVM Configuration: You must set 
-Xms
and 
-Xmx
(in 
fe.conf
via 
JAVA_OPTS
) to the same value to avoid JVM resizing overhead. 2. CN (Compute Node) Pods CNs are stateless and responsible for the actual execution of SQL queries. They are typically used in shared-data (cloud-native) deployments. * Recommended CPU: 16 Cores. * Recommended Memory: 64 GB. Parameters for Decision: * Data Volume & Query Complexity: Complex join operations, large aggregations, and window functions are CPU and memory-intensive. For heavy workloads, 32 cores and 128 GB+ RAM per pod are common. * Concurrency Requirements: StarRocks uses a pipeline execution engine. More CPU cores allow for higher parallelism per query and more concurrent queries. * Autoscaling (HPA): Since CNs are stateless, they are ideal for Horizontal Pod Autoscaling. Decisions for scaling out are typically based on: * CPU Utilization: Often set to trigger a scale-out at 60% average utilization. * Memory Utilization: Also commonly set to trigger at 60% to ensure there is enough "headroom" for large intermediate query results. Summary Table Component | CPU (Recommended) | Memory (Recommended) | Primary Driver FE | 8 Cores | 16 GB - 128 GB | Metadata size (Tablets) CN | 16 Cores | 64 GB | Query complexity & Concurrency Note for Testing/Dev: For small-scale development or testing environments, you can start as low as 1-2 CPU cores and 4-8 GB RAM, but these are not suitable for production performance. References • deployment/sr_operator.md • deployment/plan_cluster.md • quick_start/helm.md • deployment/deploy_manually.md
r
@Rocky Why I am seeing these logs in fe pods and pod is alos not coming up: 2026-03-30 052801.585Z INFO (UNKNOWN kube-starrocks-fe-0.kube-starrocks-fe-search.plattest03.svc.cluster.local_9010_1773372562608(-1)|1) [GlobalStateMgr.waitForReady():1256] wait globalStateMgr to be ready. FE type: INIT. is ready: false
r
The log message 
wait globalStateMgr to be ready. FE type: INIT. is ready: false
indicates that the Frontend (FE) process has started but has not yet completed its initialization. Specifically, it is waiting to either become the Leader or join an existing cluster as a Follower/Observer. In a Kubernetes environment, this usually happens for one of the following reasons: 1. Network Connectivity & FQDN Issues FE nodes communicate with each other using the 
edit_log_port
(default 
9010
). If they cannot reach each other, they cannot elect a leader. * Check Service Discovery: Ensure the headless service (e.g., 
kube-starrocks-fe-search
) is correctly resolving the pod FQDNs. * Check Ports: Ensure port 
9010
is open and reachable between all FE pods. 2. 
priority_networks
Configuration If the FE pod has multiple network interfaces, it might be picking up the wrong IP address (e.g., a Docker bridge IP instead of the Pod IP). * Fix: Ensure 
priority_networks
is configured in your 
fe.conf
or Helm chart values to match your Kubernetes Pod CIDR (e.g., 
10.x.x.x/16
). 3. Metadata Quorum (Election Failure) For a cluster to start, a majority (quorum) of Follower FEs must be alive. * Existing Cluster: If you have 3 FEs and 2 are down, the remaining one will stay in 
INIT
state because it cannot form a quorum. Ensure the other FE pods are also running. * New Cluster: If this is the very first time starting the cluster, the first FE (
fe-0
) should initialize. If it doesn't, check for any 
BDBJE
related errors earlier in the logs. 4. System Clock Skew StarRocks uses BDBJE for metadata replication, which is sensitive to time differences. * Check: Ensure all Kubernetes nodes have synchronized clocks (using NTP/Chrony). If the clock delta between nodes exceeds 5 seconds, the FE will fail to initialize. Look for 
Clock delta: ... exceeds max permissible delta
in the logs. 5. Metadata Corruption If the FE was killed unexpectedly, the metadata might be in an inconsistent state. * Check: Look further up in the 
fe.log
for errors like 
EnvironmentFailureException
or 
VLSN
mismatches. * Action: If this is a non-production environment and you are stuck, you might need to clear the 
meta
directory (defined by 
meta_dir
in 
fe.conf
) and restart, but do not do this on a production cluster without a backup. Next Steps to Debug: 1. Check all FE logs: Look at the logs of all FE pods, not just 
fe-0
. Often, the error causing the hang is visible in another pod. 2. Verify `fe.conf`: Check if 
JAVA_OPTS
and 
meta_dir
are correctly set. 3. Check Pod Status: Run 
kubectl get pods -o wide
to ensure all pods are on the expected network. If you see a specific error message like 
com.sleepycat.je.DatabaseNotFoundException
or 
REPLICA_ACK_TIMEOUT
, please provide those details for more specific guidance. References • administration/Meta_recovery.md • faq/Deploy_faq.md • administration/management/FE_configuration.md
r
@Rocky Only one FE and one CN pods are coming up but in pending state.
r
It sounds like your StarRocks pods are stuck at the scheduling stage. In Kubernetes, a Pending state usually means the scheduler cannot find a node that meets the pod's requirements. To diagnose this, please follow these steps: 1. Check Pod Events The most direct way to see why a pod is pending is to look at its events. Run:
Copy code
bash
kubectl describe pod <fe-pod-name>
kubectl describe pod <cn-pod-name>
Look at the 
Events
 section at the bottom. You will likely see messages like: * 
0/3 nodes are available: 3 Insufficient cpu.
* 
pod has unbound immediate PersistentVolumeClaims.
* 
node(s) had untolerated taint.
2. Common Causes & Solutions A. Insufficient Resources (Most Common) StarRocks FE and CN pods often have high default resource requests. If your nodes are small or already busy, they won't start. * Check node capacity: 
kubectl describe nodes
* Solution: Reduce the CPU/Memory requests in your StarRocksCluster YAML or Helm 
values.yaml
, or add larger nodes to your cluster. B. Persistent Volume (PVC) Issues FE pods require persistent storage for metadata. If the PVC cannot be bound, the pod stays pending. * Check PVC status: 
kubectl get pvc
* Solution: Ensure you have a StorageClass defined and available. If you are on a local cluster (like Minikube or Kind) without a dynamic provisioner, you may need to create Persistent Volumes (PVs) manually. C. Node Affinity or Taints If you have configured specific 
nodeSelector
, 
affinity
, or if your nodes have taints (e.g., `node-role.kubernetes.io/master:NoSchedule`), the pods might not have a valid place to land. * Solution: Check the 
spec
section of your StarRocksCluster resource to ensure the affinity rules match your available nodes. 3. Check the Operator Logs If the pods aren't even being created correctly, the StarRocks Operator might be encountering an error:
Copy code
bash
kubectl logs -l <http://app.kubernetes.io/name=starrocks-operator|app.kubernetes.io/name=starrocks-operator> -n <namespace>
Next Step: If you can share the output of 
kubectl describe pod <pod-name>
, I can give you a more specific solution! References • quick_start/helm.md • faq/Deploy_faq.md • deployment/deploy_manually.md • administration/Meta_recovery.md
r
@Rocky What are the limits of StarRocks broker load (file size, concurrency, throughput), and how do we ensure we don’t exceed them?
r
Broker Load is the recommended method for handling large data volumes in StarRocks, particularly from HDFS or cloud storage (S3, GCS, OSS). Unlike other methods, it is designed for scale rather than high frequency. Here are the specific limits and best practices for managing them: 1. File Size Limits * No Strict Hard Limit: Unlike Stream Load (which has a recommended limit of 10 GB per file via 
streaming_load_max_mb
), Broker Load can handle massive datasets, often in the hundreds of gigabytes or even terabytes, by splitting them into smaller scan tasks. * JSON Limit: If loading JSON, a single JSON object cannot exceed 4 GB. * Memory Constraint: The actual limit is often governed by the 
load_mem_limit
(default is 2 GB per job). If your files are very wide or have complex transformations, you may need to increase this. 2. Concurrency Limits StarRocks limits how many Broker Load jobs can run at once to protect cluster stability: * `max_broker_load_job_concurrency`: Default is 5. This specifies the maximum number of Broker Load jobs that can run concurrently in the cluster. * `max_running_txn_num_per_db`: Default is 100. This is the total transaction limit per database. * `desired_max_waiting_jobs`: Default is 1024. If you submit more jobs than the concurrency limit, they enter a 
QUEUEING
state. If this queue exceeds 1024, new requests are rejected. 3. Throughput Limits Throughput is not a fixed hard cap but depends on your hardware and cluster configuration: * Average Speed: StarRocks documentation often uses 10 MB/s per BE as a conservative estimate for calculating timeouts, though actual performance on modern hardware is typically much higher (100 MB/s+ per BE). * Parallelism: Each job is split into instances based on 
min_bytes_per_broker_scanner
(default 64 MB). The number of concurrent instances per job is 
min(Total Data / 64MB, Number of BEs)
. 4. How to Ensure You Don't Exceed Limits To avoid job failures or cluster saturation, follow these guidelines: * Adjust Timeouts: The default timeout is 4 hours. If you are loading massive files, calculate your required timeout using: 
Timeout > (Total Size / Average Load Speed)
. You can set this in the 
PROPERTIES ("timeout" = "3600")
section of your 
LOAD
statement. * Monitor Memory: If you see "Memory limit exceeded" errors, increase the 
load_mem_limit
in your 
PROPERTIES
. Also, monitor 
load_process_max_memory_limit_percent
(default 30% of BE memory) to ensure total concurrent loads don't starve the BE. * Optimize File Quantity: While Broker Load handles many files well, having thousands of tiny files (KB size) is inefficient. Aim for file sizes between 100 MB and 1 GB for optimal throughput. * Use Job Priorities (v2.5+): If your queue is full, you can assign a 
priority
to important jobs to ensure they are scheduled first once a concurrency slot opens. * Check Status: Use 
SHOW LOAD
to monitor progress. If a job is 
CANCELLED
, the 
ErrorMsg
field will usually specify if a memory or timeout limit was hit. Summary of Key Configurations: Parameter | Type | Default | Description 
max_broker_load_job_concurrency
| FE | 5 | Max concurrent Broker Load jobs 
load_mem_limit
| Job | 2 GB | Memory allowed for a single job 
timeout
| Job | 14,400s | Max time a job can run (4 hours) 
min_bytes_per_broker_scanner
| FE | 64 MB | Min data size per scan instance References • loading/StreamLoad.md • administration/management/BE_configuration.md • loading/BrokerLoad.md • sql-reference/sql-statements/loading_unloading/BROKER_LOAD.md
r
@Rocky How much time does it take for leader to be selected in fe + cn?
r
In StarRocks, the "leader" concept applies only to Frontends (FE). Compute Nodes (CN) and Backend Nodes (BE) are stateless or worker nodes that do not participate in a leader election; they simply follow the FE Leader. The time it takes to select a new FE leader depends on whether the old leader failed "gracefully" or "silently" (crashed). 1. FE Leader Election Time The FE cluster uses BDBJE (Berkeley DB Java Edition) for metadata replication and leader election. * Failure Detection (The wait time): The most significant part of the delay is detecting that the leader is gone. This is governed by the configuration parameter 
bdbje_heartbeat_timeout_second
. * Default: 30 seconds. * If a leader FE crashes or the network is cut, the surviving Follower FEs will wait for this 30-second heartbeat timeout before they conclude the leader is dead and start a new election. * Election Process (The selection time): Once the failure is detected, the election itself is extremely fast. * Duration: Usually less than 1 second. * The surviving Followers vote based on who has the most up-to-date metadata (VLSN). * Total Recovery Time: In a default setup, if a leader FE hardware fails, you can expect a total downtime of roughly 30 to 40 seconds before a new leader is active and providing services. 2. Graceful Shutdown vs. Crash * Graceful Shutdown: If you stop the Leader FE using 
stop_fe.sh
, it may notify the BDBJE group of its departure. In some cases, this can trigger a faster transition than waiting for the full 30-second timeout. * Hard Crash/Network Cut: The cluster must wait for the 
bdbje_heartbeat_timeout_second
to expire to avoid "split-brain" scenarios. 3. Role of CN Nodes during Election CN nodes (Compute Nodes) do not have a leader. Instead, they interact with the FE Leader for metadata and task scheduling. * Heartbeat Mechanism: FEs send heartbeats to CNs every 5 seconds (default). * During Election: While the FE cluster is electing a new leader, CN nodes will continue to run existing tasks if possible, but they cannot receive new queries or metadata updates until the new FE Leader is established and they successfully reconnect to it. How to tune it? If you want the leader election to be faster, you can reduce 
bdbje_heartbeat_timeout_second
in 
fe.conf
. However, be cautious: setting this too low (e.g., under 10s) can lead to "flapping" leaders if there is minor network jitter or Java GC pauses in the FE process. Summary Table: Component | Process | Duration (Default) FE | Failure Detection | ~30 seconds (
bdbje_heartbeat_timeout_second
) FE | Election Vote | < 1 second CN | Re-registration | Happens automatically once new FE Leader is up References • administration/management/FE_configuration.md • faq/operation_maintenance_faq.md • administration/Meta_recovery.md
r
@Rocky Can we add this in kube-starrocks helm chart values? We want to add configuration in fe config in configmap data: fe.conf: | label_keep_max_num=1000000 label_keep_max_second=604800
r
Yes, you can absolutely add this configuration to your 
kube-starrocks
Helm chart 
values.yaml
. In the StarRocks Kubernetes Operator (which the Helm chart uses), the standard way to provide a custom 
fe.conf
is to define a ConfigMap and then reference it in the 
starRocksFeSpec
. Here is the correct structure to add this to your 
values.yaml
file:
Copy code
yaml
starrocks:
  starrocksFESpec:
    # 1. Link the FE spec to a ConfigMap named 'fe-config'
    configMapInfo:
      configMapName: fe-config
      resolveKey: fe.conf
  
  # 2. Define the ConfigMap data within the chart
  # This will cause the chart to create a ConfigMap named 'fe-config'
  configMaps:
    - name: fe-config
      data:
        fe.conf: |
          label_keep_max_num=1000000
          label_keep_max_second=604800
Important Details: 1. 
configMapInfo
: This tells the Operator to look for a ConfigMap named 
fe-config
and use the value under the key 
fe.conf
as the configuration file for the FE. 2. 
configMaps
: This section in the Helm chart's 
values.yaml
is a helper that allows you to define the content of the ConfigMap directly. The chart will then generate a Kubernetes 
ConfigMap
resource with that data. 3. Default Configs: When you provide a custom 
fe.conf
via 
configMapInfo
, it typically overwrites the default 
fe.conf
inside the container. You should ensure your ConfigMap contains all necessary basic configurations (like ports) if the Operator doesn't merge them automatically in your version. 4. Alternative (Newer Charts): In the latest versions of the 
kube-starrocks
chart (v1.8.0+), you can often use a shortcut field directly in the spec if you don't want to define a separate ConfigMap:
Copy code
yaml
    starrocks:
      starrocksFESpec:
        config: |
          label_keep_max_num=1000000
          label_keep_max_second=604800
Note: The 
config
field approach is generally preferred in newer versions for simplicity. To Apply: After updating your 
values.yaml
, run the following to apply the changes:
Copy code
bash
helm upgrade <release_name> starrocks/kube-starrocks -f values.yaml
The FE pods will restart automatically to pick up the new configuration. You can verify the change by running 
SHOW FRONTENDS;
(though these specific labels are internal) or by checking the file inside the pod:
Copy code
bash
kubectl exec -it <fe-pod-name> -- cat /opt/starrocks/fe/conf/fe.conf
r
@Rocky What are these keys:
Copy code
label_keep_max_num
label_keep_max_second
How to check the current values of these keys?
r
These two keys control how StarRocks manages the history and metadata of your load jobs (such as Stream Load, Broker Load, Routine Load, etc.). What are these keys? 1. 
label_keep_max_num
 * Description: The maximum number of historical load job "labels" that StarRocks will keep in its metadata. * Purpose: It prevents the FE (Frontend) memory from being overwhelmed by a massive history of load jobs. * Default: 
1000
. * Behavior: If you exceed this number, StarRocks will automatically purge the oldest job records to make room for new ones, even if they haven't reached the time limit yet. 2. 
label_keep_max_second
 * Description: The maximum duration (in seconds) that the labels of completed load jobs (those in 
FINISHED
or 
CANCELLED
state) are retained. * Purpose: It ensures that historical job information is eventually cleaned up to save space. * Default: 
259200
(which is 3 days). * Behavior: Once a job is older than this value, its record is deleted from the system, and you will no longer be able to see it via commands like 
SHOW LOAD
. Why change them? Users often increase these values (as you are planning to do) when they have a very high frequency of small loads and want to be able to audit or check the status of jobs from several days ago. Your proposed values (1,000,000 labels and 7 days) are common in high-throughput production environments. ────────── How to check the current values? You can check the current values of these parameters directly via SQL using any MySQL client connected to StarRocks. Run the following command:
Copy code
sql
ADMIN SHOW FRONTEND CONFIG LIKE '%label_keep%';
The output will look like this: Key | Value | Type | IsMutable | MasterOnly | Comment 
label_keep_max_num
| 1000 | Int | Yes | No | ... 
label_keep_max_second
| 259200 | Int | Yes | No | ... Note on "IsMutable": Because these are marked as 
IsMutable = Yes
, you can actually change them temporarily without a restart by running:
Copy code
sql
ADMIN SET CONFIG ("label_keep_max_num" = "1000000");
However, using the Helm chart/ConfigMap (as you asked in the previous message) is the recommended way to make the change permanent, otherwise, the values will revert to defaults if the FE pod restarts. References • administration/management/FE_configuration.md • sql-reference/sql-statements/cluster-management/config_vars/ADMIN_SHOW_CONFIG.md
2 Views
Open in Slack
PreviousNext
Powered by