< Rocky> We re evaluating seccompProfile support in the Star StarRocks #questions-and-troubleshooting

<@U0A71G31CDV> We’re evaluating seccompProfile sup...

Rumit Jain

02/20/2026, 4:36 PM

@Rocky We’re evaluating seccompProfile support in the StarRocks Kubernetes Operator and wanted to confirm the current behavior. We see that the

StarRocksCluster

CRD schema includes

seccompProfile

under

securityContext

. However, when inspecting the actual FE/CN Pods created by the operator, we only see

fsGroup

being set in

pod.spec.securityContext

, and no

seccompProfile

propagated to either the Pod or container securityContext. I am able to deploy starrocks operator in my eks but on deploying FE and CN, i am getting issue:

Copy code

create Pod kube-starrocks-fe-0 in StatefulSet kube-starrocks-fe failed error: pods "kube-starrocks-fe-0" is forbidden: violates PodSecurity "restricted:latest": seccompProfile (pod or containers "istio-validation", "istio-proxy", "fe" must set securityContext.seccompProfile.type to "RuntimeDefault" or "Localhost")

I am using starrocks operator v 1.11.3

Rocky

02/20/2026, 4:36 PM

Hello! I'm Rocky, and I'd be happy to help you with the StarRocks Kubernetes Operator. It sounds like you've encountered a known limitation where certain versions of the StarRocks Operator define the

seccompProfile

in the CRD schema but fail to propagate that configuration down to the actual FE and CN Pod/container specifications. This is particularly problematic in EKS environments or clusters enforcing the "restricted" Pod Security Standard, which requires

seccompProfile

to be set to

RuntimeDefault

Localhost

. Why this is happening The StarRocks Operator manages the lifecycle of the cluster by generating

StatefulSets

. If the operator's internal controller logic doesn't explicitly map the

seccompProfile

field from your

StarRocksCluster

YAML into the generated

PodSpec

ContainerSpec

, Kubernetes will never see it, even if your CRD is valid. Recommended Solutions 1. Upgrade the StarRocks Operator Support for full

securityContext

propagation (including

seccompProfile

) has been a focus in recent releases to better support restricted environments. * Ensure you are using StarRocks Operator v1.9.1 or later. * Version 1.9.0+ significantly improved the mapping of

podSecurityContext

and

containerSecurityContext

fields for FE, BE, and CN components. 2. Configuration Example When using a compatible version, your

StarRocksCluster

resource should look like this (repeated for

starRocksFeSpec

starRocksBeSpec

, and

starRocksCnSpec

Copy code

yaml
apiVersion: <http://starrocks.com/v1|starrocks.com/v1>
kind: StarRocksCluster
metadata:
  name: starrockscluster-sample
spec:
  starRocksFeSpec:
    # ... other config
    podSecurityContext:
      runAsUser: 1000
      fsGroup: 1000
      seccompProfile:
        type: RuntimeDefault
    containerSecurityContext:
      allowPrivilegeEscalation: false
      capabilities:
        drop:
          - ALL
      runAsNonRoot: true
      seccompProfile:
        type: RuntimeDefault

3. Temporary Workaround: Namespace Labeling If you cannot upgrade the operator immediately, you can relax the Pod Security Standard for the specific namespace where StarRocks is deployed. This will allow the Pods to be created even without the

seccompProfile

set. Execute the following command to change the requirement from

restricted

baseline

for your namespace:

Copy code

bash
kubectl label --overwrite ns <your-namespace> \
  <http://pod-security.kubernetes.io/enforce=baseline|pod-security.kubernetes.io/enforce=baseline>

4. Verification After updating your operator or configuration, you can verify if the profile is being propagated by checking the generated StatefulSet:

Copy code

bash
kubectl get statefulset kube-starrocks-fe -o yaml | grep -A 5 seccompProfile

If you are already on the latest version and the issue persists, this may be a regression. I recommend checking the logs of the

starrocks-controller

pod to see if there are any errors during the reconciliation of your

StarRocksCluster

resource. References • deployment/sr_operator.md • deployment/helm.md

Rocky

02/20/2026, 4:51 PM

Hello! I'm Rocky, and I'd be happy to help you with the StarRocks Kubernetes Operator and your EKS deployment. It sounds like you're encountering a known issue where certain fields in the

securityContext

(specifically

seccompProfile

) are not being correctly propagated by the StarRocks Operator (v1.11.3) to the generated FE and CN Pods. This is particularly problematic in EKS environments using the "restricted" Pod Security Standard, which requires

seccompProfile

to be explicitly set. 1. Verification of Configuration In the StarRocks Operator, the

securityContext

can be defined at two levels for each component (FE, BE, CN). Please ensure you are using the

podSpec

field within your component specification. For the FE component, the structure should look like this:

Copy code

yaml
apiVersion: <http://starrocks.com/v1|starrocks.com/v1>
kind: StarRocksCluster
metadata:
  name: kube-starrocks
spec:
  starRocksFeSpec:
    # Use podSpec for Pod-level security context
    podSpec:
      securityContext:
        runAsNonRoot: true
        runAsUser: 1000
        fsGroup: 1000
        seccompProfile:
          type: RuntimeDefault
    # Use containerSecurityContext for container-level settings
    containerSecurityContext:
      allowPrivilegeEscalation: false
      capabilities:
        drop:
          - ALL
      seccompProfile:
        type: RuntimeDefault

2. Known Propagation Issue In version 1.11.3, there have been reports that the operator reconciliation logic primarily focuses on

fsGroup

and may skip the

seccompProfile

field when building the Pod spec. Recommended Actions: * Try setting both: Ensure you have the

seccompProfile

defined in both

podSpec.securityContext

and

containerSecurityContext

. The "restricted" policy often requires the profile to be visible at the container level specifically. * Upgrade the Operator: This propagation issue was identified as a blocker for Pod Security Standards (PSS) and has been addressed in newer releases. If possible, try upgrading to v1.11.4 or higher (or the latest v1.12.x branch) where PSS compliance support was significantly improved. * Check Istio sidecars: Since your error mentions

istio-proxy

and

istio-validation

, ensure your Istio injection configuration is also set to use

RuntimeDefault

for its sidecars. If the StarRocks Pod spec correctly includes a Pod-level

seccompProfile

, the sidecars should inherit it. If not, the sidecars will also trigger the violation. 3. Workaround (if an upgrade isn't immediately possible) If you cannot upgrade the operator, you may need to temporarily relax the Pod Security Standard on the namespace where StarRocks is deployed:

Copy code

bash
kubectl label namespace <your-namespace> <http://pod-security.kubernetes.io/enforce=baseline|pod-security.kubernetes.io/enforce=baseline> --overwrite

Note: This moves the security level from
restricted
to
baseline
, which is less strict but allows the pods to start while you work on the operator upgrade. If you continue to see that the fields are missing from the

kubectl get pod ... -o yaml

output even after adding them to the

StarRocksCluster

CR, please let me know, and we can look into the specific operator logs! References • deployment/sr_operator.md • faq/Others.md • deployment/environment_configurations.md • deployment/helm.md

Rumit Jain

02/20/2026, 5:47 PM

@Rocky I dont see any new version with this change

Rocky

02/20/2026, 5:47 PM

To help you find the version containing that specific change, could you please provide a bit more detail? Specifically: 1. What is the change? (e.g., a specific feature like "column renaming", a bug fix for "Routine Load", or a new SQL function) 2. Do you have a Pull Request (PR) number or an Issue number from the StarRocks GitHub repository? As a quick reference, here are some of the most recent stable releases and their key highlights: * v3.3.11: Recently released with improvements to JSON-to-Parquet exports,

AT TIME ZONE

support, and enhanced statistics collection. * v3.3.2: Introduced support for renaming columns in internal tables and reading Iceberg views. * v3.2.x / v3.1.x: These branches also receive regular maintenance releases for critical bug fixes. If the change was merged very recently into the

main

branch, it might not be in a tagged stable release yet and would likely appear in the next minor or patch version (e.g., v3.3.12 or v3.4.0). Please let me know what you are looking for, and I can check the release notes or the roadmap for you! References • release_notes/release-3.3.md • release_notes/release-3.1.md

Rumit Jain

02/20/2026, 6:45 PM

@Rocky Even after setting this:

Copy code

yaml
apiVersion: <http://starrocks.com/v1|starrocks.com/v1>
kind: StarRocksCluster
metadata:
  name: kube-starrocks
spec:
  starRocksFeSpec:
    # Use podSpec for Pod-level security context
    podSpec:
      securityContext:
        runAsNonRoot: true
        runAsUser: 1000
        fsGroup: 1000
        seccompProfile:
          type: RuntimeDefault
    # Use containerSecurityContext for container-level settings
    containerSecurityContext:
      allowPrivilegeEscalation: false
      capabilities:
        drop:
          - ALL
      seccompProfile:
        type: RuntimeDefault

seccomprifle is not coimng in the FE statefulSet /POD

Rocky

02/20/2026, 6:46 PM

It appears you are correctly defining the

seccompProfile

in your

StarRocksCluster

custom resource, but the StarRocks Operator might not be passing these specific fields down to the generated FE StatefulSet. This is often caused by one of the following reasons: 1. Operator Version Compatibility The

seccompProfile

field was introduced in Kubernetes 1.19. If you are using an older version of the StarRocks Operator, its internal API definitions (CRD) might not yet include the

seccompProfile

field in the

PodSecurityContext

SecurityContext

structs. When the operator reconciles the resource, it ignores fields it doesn't recognize. * Action: Ensure you are using the latest version of the StarRocks Operator (v1.9.0 or higher is recommended for the most complete security context support). 2. Verify CRD Support You can check if your current installation's CRD actually supports that field by running:

Copy code

bash
kubectl explain starrocksclusters.spec.starRocksFeSpec.podSpec.securityContext
# and
kubectl explain starrocksclusters.spec.starRocksFeSpec.containerSecurityContext

seccompProfile

does not appear in the output of these commands, the operator's CRD does not support it, and it will be stripped out when you apply the YAML. 3. Check for Mapping Errors In some versions of the operator, the mapping for

podSpec.securityContext

might be limited to specific fields (like

runAsUser

fsGroup

). If you find that

seccompProfile

is not supported in the CRD, you have two options: 1. Upgrade the Operator: This is the recommended path to get native support for modern K8s security features. 2. Use Annotations (Workaround for older K8s/Operators): If you are on an older Kubernetes environment or cannot upgrade the operator, you can try setting the seccomp profile via Pod annotations (though this is deprecated in newer K8s versions):

Copy code

yaml
    podSpec:
      annotations:
        <http://seccomp.security.alpha.kubernetes.io/pod|seccomp.security.alpha.kubernetes.io/pod>: "runtime/default"

4. Check Operator Logs If the fields are present in the CRD but still not appearing in the StatefulSet, check the StarRocks Operator logs for any reconciliation errors:

Copy code

bash
kubectl logs -n starrocks <starrocks-operator-pod-name>

Note: Ensure there are no typos in your YAML. In your description, you wrote "seccomprifle," but in the YAML snippet, it is correctly spelled

seccompProfile

. If the spelling in the actual file is incorrect, the field will be ignored. References • quick_start/helm.md • deployment/sr_operator.md

Rumit Jain

02/20/2026, 6:47 PM

@Rocky How to set secruotyContexts in kube-starrocks-initpwd job?

Rocky

02/20/2026, 6:48 PM

To set the

securityContext

for the

kube-starrocks-initpwd

job when using the StarRocks Helm chart or Operator, you can configure it within the

initPassword

section of your configuration. 1. Using the Helm Chart (
values.yaml
) In the

kube-starrocks

Helm chart, you can define both

podSecurityContext

and

containerSecurityContext

under the

starrocks.initPassword

key. This allows you to comply with restricted Pod Security Standards or specific runtime requirements. Add the following to your `my-values.yaml`:

Copy code

yaml
starrocks:
  initPassword:
    enabled: true
    passwordSecret: starrocks-root-pass
    # Set the securityContext for the container
    containerSecurityContext:
      runAsUser: 1000
      allowPrivilegeEscalation: false
      capabilities:
        drop:
          - ALL
    # Set the securityContext for the pod
    podSecurityContext:
      runAsNonRoot: true
      fsGroup: 1000

2. Using the StarRocks Operator (
StarRocksCluster
CRD) If you are defining the

StarRocksCluster

custom resource directly, the structure is similar. Ensure the fields are placed under the

initPassword

specification:

Copy code

yaml
apiVersion: <http://starrocks.com/v1|starrocks.com/v1>
kind: StarRocksCluster
metadata:
  name: starrockscluster-sample
spec:
  initPassword:
    enabled: true
    passwordSecret: starrocks-root-pass
    containerSecurityContext:
      runAsUser: 1000
    podSecurityContext:
      runAsNonRoot: true
  starrocksFeSpec:
    # ... FE spec ...
  starrocksBeSpec:
    # ... BE spec ...

Key Considerations: * Password Secret: Ensure the secret defined in

passwordSecret

already exists in the same namespace before the job runs. * Job Behavior: The

kube-starrocks-initpwd

pod often goes through

CrashLoopBackOff

Error

states initially while waiting for the FE and BE nodes to become ready. This is expected behavior until the MySQL port is reachable and the password can be set. * Privileges: The job requires enough permissions to connect to the StarRocks cluster via the MySQL protocol (usually port 9030) to execute the

SET PASSWORD

command. If you are using an older version of the Helm chart that does not support these specific fields, you may need to update to the latest version of the

kube-starrocks

chart or the StarRocks Kubernetes Operator. References • quick_start/helm.md • loading/Spark-connector-starrocks.md • deployment/post_deployment_setup.md • administration/user_privs/authentication/native_authentication.md

Rumit Jain

02/21/2026, 9:59 AM

@Rocky From where can i download starrocks 4.0.6 cluster version helm templates? Also why there are two app versions 4.0.6 and 3.5-latest

Rocky

02/21/2026, 9:59 AM

To download the StarRocks Helm templates and understand the versioning, here is the information you need: 1. Where to download StarRocks Helm Templates? The official Helm templates for StarRocks are maintained in the starrocks-kubernetes-operator GitHub repository. You can find them here: * Helm Charts Directory: StarRocks Kubernetes Operator - Helm Charts * Artifact Hub: kube-starrocks on Artifact Hub To add the official repository and search for the templates locally, use:

Copy code

bash
helm repo add starrocks <https://starrocks.github.io/starrocks-kubernetes-operator>
helm repo update
helm search repo starrocks -l

2. Why are there two versions (4.0.6 and 3.5-latest)? In the context of Helm charts, you are likely seeing a distinction between the Chart Version and the App Version: * Chart Version (e.g., 4.0.6): This refers to the version of the Helm chart itself. It tracks changes to the Kubernetes manifests, templates, and the logic used to deploy the cluster. A new chart version is released whenever the deployment configuration (like

values.yaml

structure or resource definitions) is updated, even if the StarRocks software version remains the same. * App Version (e.g., 3.5-latest): This refers to the version of the StarRocks software (the FE and BE images) that the chart will deploy by default. * 3.5-latest is a rolling tag used by the community to point to the most recent stable patch release of the StarRocks 3.5.x minor branch. * Using a

-latest

tag ensures you get the latest bug fixes for that specific branch without manually updating the image tag every time a patch is released. Note: As of now, StarRocks is in the 3.x major release series (e.g., 3.3, 3.4, 3.5). There is no "StarRocks 4.0" application version yet; the

4.0.6

you see is almost certainly the Helm Chart version. Summary Table Version Type | Example | Description Chart Version | 4.0.6 | The version of the deployment templates/scripts. App Version | 3.5-latest | The version of the actual StarRocks database being installed. When installing, you can override the

App Version

in your

values.yaml

if you need a specific patch version (e.g.,

3.5.1

) instead of the rolling

3.5-latest

tag. References • quick_start/helm.md • deployment/helm.md

Rumit Jain

02/21/2026, 10:01 AM

@Rocky Is this not the starrocks app version 4.0.6? https://github.com/StarRocks/starrocks/tree/4.0.6

Rocky

02/21/2026, 10:01 AM

Based on the current official documentation and release history, there is no official version 4.0.6 for the StarRocks core engine. As of early 2025, the StarRocks versioning is in the 3.x series. Specifically: * The latest stable minor version is 3.3.x (with recent releases like 3.3.13). * Previous major versions include 3.0.x, 2.5.x, and 2.0.x. * There are versions 3.0.6 (released in September 2023) and 2.0.6 (released in May 2022), which follow a similar numbering pattern. The GitHub link you provided (`https://github.com/StarRocks/starrocks/tree/4.0.6`) might lead to a 404 error or a non-existent branch/tag unless it is a very specific development branch or an experimental tag that has not been officially released to the public as a stable version. If you are looking for the latest stable release to use in production, I recommend checking the official StarRocks Release Notes or the GitHub Releases page for the most current

v3.3.x

v3.2.x

tags. References • deployment/sr_operator.md • faq/shared_data_faq.md • deployment/helm.md

Rumit Jain

02/21/2026, 10:17 AM

@Rocky Is StarRocks 4.0.6 officially supported with operator v1.11.4?

Rocky

02/21/2026, 10:17 AM

Hello! I am Rocky. Based on the official StarRocks documentation and current release history, StarRocks 4.0.6 and StarRocks Operator v1.11.4 are not yet officially released versions. As of early 2025, here is the current status of the official versions: * StarRocks: The latest major stable versions are in the 3.x series. Specifically, StarRocks v3.3.x is the current stable line, and v3.4.0 was recently released (January 2024/2025). There is currently no official release of a 4.0 version. * StarRocks Operator: The official Kubernetes Operator version is currently in the v1.9.x to v1.10.x range. It is possible you might be looking for a different component or a version of another software. If you meant StarRocks 3.0.6 or a different version, please let me know, and I can check the compatibility matrix for you! For the most up-to-date compatibility information, I recommend checking the StarRocks Kubernetes Operator GitHub repository or the StarRocks Release Notes. References • release_notes/release-2.2.md • sql-reference/sql-statements/table_bucket_part_index/SELECT.md • release_notes/release-2.5.md • data_source/External_table.md • faq/Sql_faq.md • sql-reference/sql-functions/json-functions/json-operators.md

Rumit Jain

04/14/2026, 4:47 AM

@Rocky What is the recommended way to create users, their roles and privileges once the cluster is up using helm chart.

Rocky

04/14/2026, 4:48 AM

When deploying StarRocks using the Helm chart (specifically

kube-starrocks

), the recommended approach for managing users, roles, and privileges involves a combination of automated root initialization and standard SQL-based administration. 1. Initialize the
root
User Password The first step is securing the default

root

account. The Helm chart provides a built-in mechanism to set this during the initial deployment. * Create a Kubernetes Secret: Before installing the chart, create a secret containing your desired password.

Copy code

bash
    kubectl create secret generic starrocks-root-pass --from-literal=password='your_secure_password'

* Configure `values.yaml`: Enable the

initPassword

feature in your Helm configuration.

Copy code

yaml
    starrocks:
      initPassword:
        enabled: true
        passwordSecret: starrocks-root-pass

When you run

helm install

, the StarRocks Operator will launch a temporary job (

kube-starrocks-initpwd

) that connects to the FE and applies the password. 2. Create Users and Roles (RBAC) Since there is currently no native Kubernetes Custom Resource (CRD) for managing individual StarRocks users or roles, you must manage them using Role-Based Access Control (RBAC) SQL commands once the cluster is up. Recommended Workflow: 1. Connect to the cluster: Use a MySQL client via the FE service (Port 9030). 2. Create Roles: Group privileges into roles rather than granting them directly to users.

Copy code

sql
    CREATE ROLE developer_role;
    GRANT SELECT, INSERT ON TABLE mydb.* TO ROLE developer_role;

3. Create Users and Assign Roles:

Copy code

sql
    CREATE USER 'alice'@'%' IDENTIFIED BY 'password123';
    GRANT developer_role TO USER 'alice'@'%';
    SET DEFAULT ROLE developer_role TO 'alice'@'%';

3. Recommended for Production: External Authentication For production environments, the "recommended way" to manage users at scale is to avoid creating local users manually. Instead, configure StarRocks to use LDAP or Microsoft Active Directory. You can configure this by adding the relevant parameters to your FE configuration via the Helm `values.yaml`:

Copy code

yaml
starrocks:
  starrocksFESpec:
    configMapInfo:
      configMapName: fe-config
      resolveKey: fe.conf

In your

fe.conf

, you would include LDAP settings (e.g.,

ldap_authentication_enabled = true

). Once enabled, StarRocks can automatically authenticate users against your corporate directory, and you only need to manage Roles and Grants locally. 4. Automation via Kubernetes Jobs If you need to automate the creation of specific users or schemas during a CI/CD pipeline (e.g., via ArgoCD), the standard pattern is to deploy a Kubernetes Job alongside your Helm release. This job should: 1. Use a

mysql:client

image. 2. Wait for the FE service to be ready (

initContainers

or a wait script). 3. Execute a script containing your

CREATE USER

and

GRANT

statements. Summary of Best Practices Task | Recommended Method Initial Security | Use

initPassword

in Helm values with a K8s Secret. User/Role Management | Use standard SQL (

CREATE USER

GRANT

) via a MySQL client. Scalable Auth | Configure LDAP/AD in

fe.conf

to avoid manual user management. Automation | Use a post-install Kubernetes Job to run initialization SQL scripts. References • quick_start/helm.md • deployment/helm.md • quick_start/shared-nothing.md

Rumit Jain

04/17/2026, 12:51 PM

@Rocky Suddenly my FE pods stared getting down with OOM. What to debug?

Rocky

04/17/2026, 12:51 PM

When your StarRocks FE pods are experiencing OOM (Out of Memory) issues in a Kubernetes environment, the debugging process follows two main paths: Kubernetes-level OOM (the container was killed by the OS/Kubelet) and JVM-level OOM (the Java process crashed or exited due to memory exhaustion). 1. Identify the Type of OOM First, determine if the pod was killed by Kubernetes or if the application crashed internally. * Check Pod Status:

Copy code

bash
    kubectl describe pod <fe-pod-name>

Look for the

Last State: Terminated

section. * Reason: OOMKilled: This means the process exceeded the Kubernetes memory limit defined in your Helm

values.yaml

. * Reason: Error (Exit Code 137 or 1): This often indicates the JVM itself crashed or the FE process exited because of a "Full GC" timeout or internal

OutOfMemoryError

. 2. What to Debug & Collect Once you know the "where," look into the "why": A. Check Logs (JVM/Application Level) * fe.out & fe.log: Search for

java.lang.OutOfMemoryError

. * Leader Switch: Look for

transfer FE type from LEADER to UNKNOWN. exit

. If the FE is a Leader and experiences a long "Stop-the-World" GC pause (usually >30s), it will voluntarily exit to allow another node to take over. This is a common cause of "crashes" that look like OOM. * GC Logs: If you have GC logging enabled, check for frequent "Full GC" events. B. Analyze Memory Distribution (Inside the Pod) If the pod stays up long enough, exec into it:

Copy code

bash
kubectl exec -it <fe-pod-name> -- bash

* Live Histogram: See which objects are taking up space without a full dump.

Copy code

bash
    jmap -histo:live <pid> | head -n 20

* Heap Dump: For deep analysis (caution: this will freeze the FE for several seconds/minutes).

Copy code

bash
    jmap -dump:live,format=b,file=/opt/starrocks/fe/log/dump.hprof <pid>

* Memory Profiles (v3.3.6+): StarRocks now automatically generates memory allocation flame graphs in

fe/log/proc_profile/

. Check these

.tgz

files to see which functions are allocating the most memory. 3. Common Causes & Fixes Cause 1: Heap Size vs. Container Limit (K8s Specific) The most common mistake is setting the JVM Heap (

-Xmx

) too close to the Kubernetes memory limit. * Rule of thumb: Your

-Xmx

should be 75% to 80% of the container's memory limit. The remaining 20% is needed for off-heap memory, thread stacks, and system overhead. * Fix: Update your `values.yaml`:

Copy code

yaml
    starrocksFESpec:
      resources:
        limits:
          memory: 16Gi  # K8s Limit
      configMapInfo:
        resolveKey: fe.conf
        configMapName: fe-config # In your fe.conf, set JAVA_OPTS="-Xmx12G -Xms12G"

Cause 2: Metadata Growth (Tablets) FEs store all metadata (tablet locations, etc.) in memory. If you have millions of tablets, the FE will require more memory. * Check: Run

SHOW FRONTENDS;

and check the

Records

Tablets

count in your cluster. * Recommendation: * < 1M tablets: 16GB RAM * 1M - 2M tablets: 32GB RAM Cause 3: Off-Heap Memory Leak If the JVM heap is low but the pod is still

OOMKilled

, it’s likely an off-heap issue (e.g., glibc memory arenas). * Fix: Add

MALLOC_ARENA_MAX=1

to the environment variables in your FE spec or

fe.conf

. This prevents glibc from creating too many memory pools. Cause 4: Heavy Query Execution In recent versions, FEs can consume significant memory when generating complex query plans for very large schemas. * Fix: If you are on v3.3.10+, check the

QueryFEAllocatedMemory

metric in your logs to identify memory-heavy queries. Summary Checklist 1. [ ] Is the pod status

OOMKilled

? (Increase K8s memory limit). 2. [ ] Is

-Xmx

set to 80% of the limit? (Adjust

JAVA_OPTS

). 3. [ ] Are there frequent Full GCs in

fe.log

? (Check tablet count / metadata size). 4. [ ] Did you add

MALLOC_ARENA_MAX=1

? (Recommended for all K8s deployments). References

Rocky

04/17/2026, 12:51 PM

• quick_start/helm.md • faq/Others.md • sql-reference/sql-functions/JAVA_UDF.md • knowledge/trouble_shooting/fe_memory_problems.md • administration/management/BE_configuration.md

Rumit Jain

04/17/2026, 12:58 PM

@Rocky Seeing this: 2026-04-17 125431.973Z WARN (UNKNOWN kube-starrocks-fe-2.kube-starrocks-fe-search.plattest03.svc.cluster.local_9010_1773372610386(-1)|1) [GlobalStateMgr.waitForReady():1261] It took too much time for FE to transfer to a stable state(LEADER/FOLLOWER), it maybe caused by one of the following reasons: 1. There are too many BDB logs to replay, because of previous failure of checkpoint(you can check the create time of image file under meta/image dir). 2. Majority voting members(LEADER or FOLLOWER) of the FE cluster haven't started completely. 3. FE node has multiple IPs, you should configure the priority_networks in fe.conf to match the ip record in meta/image/ROLE. And we don't support change the ip of FE node. Ignore this reason if you are using FQDN. 4. The time deviation between FE nodes is greater than 5s, please use ntp or other tools to keep clock synchronized. 5. The configuration of edit_log_port has changed, please reset to the original value. 6. The replayer thread may get stuck, please use jstack to find the details.

Rocky

04/17/2026, 12:58 PM

The warning you are seeing is a direct symptom of the FE failing to reach a stable state, and in your case, it is likely tied to the OOM (Out of Memory) issue you reported earlier. When an FE starts up, it must replay all the journals (BDB logs) since the last "checkpoint" (the last time the metadata was saved to an

image

file). If the FE is OOM-ing, it's often because the volume of journals it needs to load into memory is exceeding the available heap/container memory. Why this is happening: 1. Metadata Accumulation: If your FEs have been crashing or under high memory pressure, they may have stopped performing "checkpoints." Without checkpoints, the BDB logs grow indefinitely. 2. Startup Memory Spike: During startup, replaying 100,000+ journals requires significantly more memory than a running FE. If your

-Xmx

or K8s memory limit is too tight, the pod will OOM before it ever becomes "Ready." 3. Quorum Failure: In a Helm deployment, if 2 out of 3 FEs are OOM-ing, the remaining FE cannot form a majority and will stay in the

UNKNOWN

state indefinitely, printing that exact warning. Step-by-Step Debugging & Resolution 1. Check the Image Age (Metadata Health) Exec into one of the FE pods (even if it's not "Ready"):

Copy code

bash
ls -lh /opt/starrocks/fe/meta/image/

* Look at the date of the
image.<version>
files. * If the latest image is more than a few hours old (and you have a busy cluster), checkpoints are failing. _ If you see thousands of

je.info._

or BDB log files, the FE is struggling to replay them. 2. Increase Memory Temporarily If the FE is OOM-ing during the "Replay" phase, you must increase the memory limits in your

values.yaml

to allow it to finish the replay and create a new checkpoint. * Increase

resources.limits.memory

. * Increase

-Xmx

fe.conf

(aim for 80% of the limit). 3. Resolve "Majority" Issues (K8s) If your cluster is stuck because FEs can't find each other: * Check FQDNs: Ensure the FE pods can ping each other using their internal K8s service names (e.g.,

kube-starrocks-fe-0.kube-starrocks-fe-search...

). * Wipe "Bad" Nodes (Last Resort): If one FE has corrupted metadata or is too far behind, you may need to delete its Persistent Volume Claim (PVC) to force it to perform a Network Recovery (downloading a fresh metadata image from the Leader). _ _Warning: Only do this for one follower at a time, never the Leader.* 4. Address the Disk/IO Latency The log mentions

LogbufferPool_FullLatch

. This happens when the disk where

meta_dir

is stored is too slow. * Ensure your FE PVCs are using High-Performance SSDs (e.g.,

gp3

on AWS,

premium-ssd

on Azure). * Slow disk IO causes the BDB journal writing to back up, leading to memory pressure and eventually OOM. 5. Recommended
fe.conf
adjustments for K8s In your Helm

values.yaml

, ensure these are set to prevent common OOM/Consensus issues:

Copy code

yaml
fe_configs:
  # Reduce memory overhead
  JAVA_OPTS: "-Xmx8g -Xms8g -XX:+UseG1GC"
  # Mandatory for K8s to prevent glibc memory bloat
  MALLOC_ARENA_MAX: "1"
  # Increase the threshold for manual checkpointing if needed
  edit_log_roll_num: "50000"

Summary of the "Fix" 1. Give it more RAM: The OOM is preventing the "Stable State." Raise the K8s limit. 2. Check Disk Space: Ensure the

meta_dir

volume is not 100% full (FEs need >5GB free to function). 3. Wait: Once memory is increased, the first startup might take 5–10 minutes as it replays the backlog of logs. Monitor

fe.log

to see the replay progress. References • administration/Meta_recovery.md • administration/management/FE_configuration.md

Rumit Jain

04/17/2026, 1:10 PM

@Rocky How to check what is causing this?

Rocky

04/17/2026, 1:10 PM

To pinpoint why your FE is stuck in an

UNKNOWN

state or crashing with OOM, follow this diagnostic checklist. These steps progress from basic log analysis to deeper JVM inspection. 1. Check for "Metadata Lag" or "Replay Storm" The warning message explicitly mentions "too many BDB logs to replay." If the FE has been down or OOM-ing for a while, it may have a massive backlog of journals to load into RAM. * Check BDB log count: Exec into the FE pod and count the

.jdb

files:

Copy code

bash
    ls -1 /opt/starrocks/fe/meta/bdb | wc -l

* Healthy: Usually fewer than 10-20 files. * Problematic: If you see hundreds or thousands of files, the FE is likely failing to perform a "checkpoint." Loading these files into memory during startup is the most common cause of OOM at boot time. * Check Image Age:

Copy code

bash
    ls -lh /opt/starrocks/fe/meta/image

Look for the latest

image.<id>

file. If it is older than 24 hours, the CheckpointThread is stuck or failing, which is why the BDB logs are piling up. 2. Inspect the "Memory Profiles" (Available in v3.3.6+) StarRocks now automatically dumps memory allocation flame graphs when memory pressure is high. * Locate the files:

Copy code

bash
    ls /opt/starrocks/fe/log/proc_profile/

* How to read: Look for files ending in

.tgz

. Download them to your local machine and open the HTML inside. * What to look for: A wide frame for

BDBEnvironment.getDatabaseNamesWithPrefix

Metadata.loadImage

indicates the OOM is metadata-related. A wide frame for

QueryPlanner

indicates query-load-related OOM. 3. Lightweight JVM Inspection If the pod is currently running but "Not Ready," run these commands immediately: * Object Histogram: See which Java objects are eating the heap without crashing the process.

Copy code

bash
    # Replace <pid> with the actual process ID, usually 1 or found via 'ps -ef'
    jmap -histo:live <pid> | head -n 30

_ _Frequent culprits:*

org.apache.starrocks.catalog.Tablet

byte[]

String

. * Thread Dump (jstack): The log suggests the replayer thread might be stuck.

Copy code

bash
    jstack <pid> > /opt/starrocks/fe/log/fe_stack.txt

Search the output for

replayer

. if it's stuck on a lock or disk IO, that’s your bottleneck. 4. Search
fe.log
for Specific Fatal Errors Grep for these specific keywords that indicate why the "stable state" isn't being reached: * INSUFFICIENT_LOG
: This node is too far behind the leader and needs a "Network Recovery" (it will never start on its own). * ReplicaWriteException
: Indicates a version mismatch in BDBJE logs between nodes. * transfer FE type from LEADER to UNKNOWN
: This means the FE was a leader but lost its heartbeat to the majority. * HANDSHAKE_ERROR
: Likely an IP or

priority_networks

configuration issue where nodes can't agree on which network interface to use. 5. Verify the "Quorum" status If you can connect to the MySQL port (9030) of any FE that is alive:

Copy code

sql
SHOW PROC '/frontends'\G

* Check
Alive
: If multiple nodes are

false

, the cluster has no quorum. * Check
ReplayedJournalId
: If one node has a much lower ID than the others, it is stuck replaying logs. Summary of what to "Fix" based on findings: * If BDB logs are huge (>500 files): You must temporarily double the RAM in your Helm chart to let the FE finish replaying and "checkpointing." Once the file count drops, you can lower the RAM again. * If you see
OOMKilled
in K8s but no OOM in Java logs: You likely have a "Direct Buffer" or "Glibc" leak. Ensure

MALLOC_ARENA_MAX=1

is set in your environment variables. * If you see "Majority not started": Ensure all 3 FE pods (if you have 3) are at least attempting to start. If one has a corrupted PVC, it can prevent the other two from ever becoming LEADER/FOLLOWER. References

Rocky

04/17/2026, 1:10 PM

• administration/management/logs.md • deployment/deploy_manually.md • administration/management/FE_configuration.md • faq/operation_maintenance_faq.md • knowledge/trouble_shooting/trouble_shooting_scripts.md

Rumit Jain

04/17/2026, 1:55 PM

@Rocky I’m running StarRocks FE on Kubernetes (via Helm) and seeing repeated OOMKills leading to FE restarts and inconsistent warehouse state. Current configuration FE pod resources: • Requests: 8Gi • Limits: 8Gi JAVA_OPTS:

Copy code

-Dlog4j2.formatMsgNoLookups=true 
-Xmx8192m 
-XX:+UseG1GC 
-Xlog:gc*:${LOG_DIR}/fe.gc.log.$DATE:time

Observed behavior FE pods getting

OOMKilled

(Exit Code 137) • Frequent restarts (

CrashLoopBackOff

) • Intermittent query failures and connection drops •

SHOW WAREHOUSES

sometimes shows

NodeCount = 0

even though compute nodes are alive Questions Is it expected that

-Xmx

equal to container memory (8Gi) leads to OOM in StarRocks FE due to JVM overhead? What is the recommended ratio between: container memory 1. JVM heap (

-Xmx

) 2. For production workloads, what is a safe baseline for FE sizing (memory + heap)? 3. Are there known memory-heavy components in FE (e.g., metadata, tablets, query planning) that should influence sizing? 4. Any recommended GC tuning or JVM flags for FE stability? Context • StarRocks version: 4.0.1 • Running on EKS • Moderate dataset (few hundred tablets per CN currently)

Rocky

04/17/2026, 1:55 PM

Your current configuration is the primary cause of the

OOMKilled

(Exit Code 137) events. In a Kubernetes environment, setting the JVM heap size (

-Xmx

) equal to the container memory limit is a guaranteed path to failure. 1. Why
-Xmx
= Container Limit leads to OOM Yes, this is absolutely expected. The JVM consumes memory beyond just the "Heap." When you set

-Xmx8192m

and a container limit of

8Gi

, you leave zero bytes for the following critical non-heap memory areas: * Metaspace: Stores class metadata. * Code Cache: Stores JIT-compiled code. * Thread Stacks: Each Java thread typically takes ~1MB. FE nodes can have hundreds of threads. * Direct Buffers: Used for high-performance network I/O (essential for FE-BE communication). * Native Overhead: Memory used by the OS and glibc (native memory allocators). The moment the JVM attempts to allocate memory for any of these, the total container usage exceeds 8Gi, and the Kubernetes Kubelet immediately sends a

SIGKILL

. 2. Recommended Ratios and Baselines For StarRocks FE pods in Kubernetes, use the following rules of thumb: * Heap Ratio: Set

-Xmx

to 75% to 80% of the container's memory limit. _ _For your 8Gi limit:* Set

-Xmx6g -Xms6g

. * Production Baseline: For production workloads, the safe baseline is 16Gi RAM and 8 vCPUs. * StarRocks documentation recommends 16GB minimum to support up to 1 million tablets. * If your metadata grows (more tables/partitions), you should scale to 32Gi or 64Gi. 3. Memory-Heavy Components in FE Several components consume significant memory and should influence your sizing: * Tablet Metadata: Every tablet's location and version info is stored in the FE's RAM. If you have many partitions or small buckets, your memory requirements will spike. * BDBJE Logs: During startup, the FE replays Berkeley DB journals. Replaying a large backlog of logs is memory-intensive and is a common cause of OOM during the boot phase. * Query Planning: Complex SQL queries with many joins or a large number of table schemas can cause transient memory spikes in the FE's query planner. * Image Checkpointing: The process of saving metadata to a disk "image" uses a separate memory buffer. (In v3.4+, this is offloaded to followers to protect the Leader, but it still requires memory on the node performing the task). 4. Recommended JVM & GC Tuning for Stability To stabilize your FE pods on EKS, update your

JAVA_OPTS

and environment variables: Environment Variable (Critical) Add this to your FE deployment spec (it is often the "missing link" for K8s OOM issues):

Copy code

bash
MALLOC_ARENA_MAX=1

Why? By default, glibc creates multiple memory pools (arenas) for multi-threaded processes. This can cause the FE process's virtual and resident memory to bloat significantly. Setting this to

limits the number of arenas and prevents off-heap memory fragmentation. Improved
JAVA_OPTS
Assuming a 16Gi container limit:

Copy code

bash
-Xmx12g 
-Xms12g 
-XX:+UseG1GC 
-XX:MaxGCPauseMillis=200 
-XX:+UnlockDiagnosticVMOptions 
-XX:+G1SummarizeConcMark 
-Xlog:gc*

Note: Always set
-Xms
and
-Xmx
to the same value to prevent the JVM from constantly resizing the heap, which causes latency. Why
NodeCount = 0
? This is a side effect of the OOM crashes. When the Leader FE is killed, the cluster must elect a new Leader. If the pods are in a

CrashLoopBackOff

, the FEs never reach a "Stable State" (LEADER/FOLLOWER). Until an FE becomes a stable Leader and completes its handshake with the Compute Nodes (CNs), it will report 0 active nodes for the warehouse. Summary Checklist: 1. Increase Resources: Change FE limits to at least

16Gi

. 2. Adjust Heap: Set

-Xmx12g

(if using 16Gi limit). 3. Set Environment Var: Add

MALLOC_ARENA_MAX=1

. 4. Check Tablet Count: Run

SHOW PROC '/statistic';

to see if your tablet count is unexpectedly high, which might require even more RAM. References

Rocky

04/17/2026, 1:55 PM

• deployment/environment_configurations.md • knowledge/trouble_shooting/fe_memory_problems.md • administration/management/FE_configuration.md • quick_start/helm.md • administration/management/BE_configuration.md • faq/Others.md • deployment/plan_cluster.md

Rumit Jain

04/17/2026, 1:57 PM

@Rocky This is my config:

Copy code

config: |
    run_mode = shared_data
    cloud_native_storage_type = S3
    aws_s3_path = k-starrocks-data
    aws_s3_region = us-east-1
    aws_s3_endpoint = <https://s3.us-east-1.amazonaws.com>
    enable_load_volume_from_conf=true
    aws_s3_use_instance_profile=false
    aws_s3_use_aws_sdk_default_behavior=true
    enable_trace_historical_node = true
    audit_log_modules = slow_query, query, connection
    audit_log_json_format = true
    qe_slow_log_ms = 1000
    aws_s3_enable_partitioned_prefix=true
    aws_s3_num_partitioned_prefix=64
    automated_cluster_snapshot_interval_seconds=600
    label_keep_max_num=1000000
    label_keep_max_second=604800

Rocky

04/17/2026, 1:59 PM

Your configuration contains a "smoking gun" that explains why your FE is OOM-ing, especially with such a small (8Gi) memory limit. The Major Culprit: Label Retention In your

fe.conf

, you have set:

Copy code

yaml
label_keep_max_num=1000000
label_keep_max_second=604800 (7 days)

This is the primary cause of your OOM. Every time you perform a load (Stream Load, Broker Load, Routine Load, etc.), StarRocks stores the "Label" and the job's metadata in the FE's memory to prevent duplicate submissions and provide history. * The Problem: Keeping 1,000,000 labels in memory for 7 days can consume several gigabytes of Java heap. * The Impact: With only 8Gi of total memory (and 8Gi assigned to

-Xmx

), the JVM has no room for this metadata plus the actual cluster metadata (tablets, schemas). When the FE starts up and replays the BDB logs, it tries to load all 1,000,000 labels into the heap at once, causing the

OOMKilled

crash before it can ever reach a "Ready" state. Recommendation: Reduce these to more sustainable levels unless you have a specific legal/business requirement to see 7 days of load history via

SHOW LOAD

Copy code

yaml
label_keep_max_num = 50000 # Default is 1000
label_keep_max_second = 86400 # 24 hours

────────── Other Shared-Data Specific Impacts Since you are running in

shared_data

mode on S3: 1. automated_cluster_snapshot_interval_seconds=600
: Taking a cluster snapshot every 10 minutes is quite frequent. While snapshots are essential in shared-data mode, the process of serializing metadata to S3 creates transient memory spikes. With your current 0-byte margin (8Gi heap in an 8Gi container), this spike will trigger an immediate K8s OOMKill. 2. Metadata Overhead: In shared-data mode, the FE manages more complex versioning and tablet metadata for S3. This requires more "headroom" in the non-heap memory area for internal buffers. ────────── Revised Strategy to Fix Your Cluster 1. Immediate Config Fix (Helm
values.yaml
) Update your

config

block to reduce the label pressure and fix the JVM ratio:

Copy code

yaml
fe_configs:
  label_keep_max_num: "50000"
  label_keep_max_second: "86400"
  # Add this to help with K8s memory management
  MALLOC_ARENA_MAX: "1"

# Increase your resources! 8Gi is not enough for a production-like FE
resources:
  limits:
    memory: 16Gi
  requests:
    memory: 16Gi

# Adjust JAVA_OPTS to provide 25% overhead for non-heap/OS
# (Assuming 16Gi limit)
JAVA_OPTS: "-Xmx12g -Xms12g -XX:+UseG1GC"

2. If the FEs are stuck in
CrashLoopBackOff
If the FEs won't even start to accept the new config because they are replaying the old "bloated" BDB logs: 1. Scale down the FE statefulset to 0. 2. Temporarily increase the memory limit to something very high (e.g., 32Gi) and

-Xmx24g

. 3. Scale up to 1. This gives the FE enough "breathing room" to finish replaying the 1,000,000 labels, perform a Checkpoint (which saves the state to an

image

file), and then clean up the old BDB logs. 4. Once the FE is healthy, apply the lower

label_keep_max_num

and you can scale the memory back down to 16Gi. Summary * Heap vs Limit: Never set

-Xmx

equal to the container limit. * Label Bloat: Your

label_keep_max_num

is 1000x the default; this is likely occupying 4-6Gi of your 8Gi heap. * Recommendation: Use 16Gi RAM / 12Gi Heap and reduce label retention. References • administration/management/FE_configuration.md

Rumit Jain

04/17/2026, 4:36 PM

@Rocky How to confirm that these values are the reason of OOM?

Copy code

label_keep_max_num=1000000
label_keep_max_second=604800 (7 days)

I updated these values to lower, restarted the fe pods and again updated back to these values and resatrted the fe pods. Now they are healthy but how i can confirm that these values are too high?

Rocky

04/17/2026, 4:36 PM

To confirm that your label retention settings were the cause of the OOM, you can look for evidence in three places: the FE internal logs, the system process statistics, and the StarRocks internal procedures. Since you've already lowered the values and restarted, the "bloated" state is gone, but you can verify the current impact of your settings and cross-reference with historical logs. 1. Check the FE
MemoryUsageTracker
Logs StarRocks FEs (v3.2+) have a background thread that periodically calculates the memory footprint of different metadata modules. * Action: Search your

fe.log

for the "Load" module's memory estimation. * Command:

Copy code

bash
    grep "Module Load - LoadMgr" fe.log

* What to look for: You will see a line like this: >

INFO ... Module Load - LoadMgr estimated 2.4GB of memory. Contains LoadJob with 850000 object(s).

If the

estimated

value was several gigabytes and the

object(s)

count was approaching your previous limit of 1,000,000, that is your "smoking gun." 2. Use
SHOW PROC
to Count Metadata You can see exactly how many finished jobs are currently being held in memory across your databases. * SQL Command:

Copy code

sql
    SHOW PROC '/jobs';

* Deep Dive: To see the counts for a specific database (replace

<db_id>

with the ID from the previous command):

Copy code

sql
    SHOW PROC '/jobs/<db_id>';

* Verification: Check the

Finished

column. If the sum of finished jobs across all databases was near 1,000,000, it confirms the FE was forced to keep those labels (and their associated metadata strings) in the Java Heap. 3. Analyze the JVM Object Histogram Even if the pods are currently healthy, you can see how much space a single LoadJob occupies. * Action: Exec into the FE pod and run `jmap`:

Copy code

bash
    # Replace <pid> with the actual process ID, usually 1
    jmap -histo:live <pid> | grep -E "LoadJob|StreamLoadTask|Label"

* Calculation: Multiply the "Bytes" value by the ratio of your previous limit vs. current limit. Example: If 50,000

LoadJob

objects take 200MB, then 1,000,000 objects will take ~4GB. Adding the overhead of

String

labels and metadata, this easily exhausts an 8Gi container limit. 4. Check Historical "Memory Profiles" If the FE crashed or experienced high pressure, StarRocks (v3.3.6+) automatically dumps a profile. * Path: Look in

/opt/starrocks/fe/log/proc_profile/

. * Analysis: If you see files from the time of the OOM, download and open the

.tgz

(HTML inside). Look for a wide bar in the flame graph containing

LoadMgr

LabelMgr

, or

editlog.replay

. Summary of why it "felt" healthy after the second update: When you lowered the values and restarted, the FE deleted the old labels from its memory and metadata. When you set the values back to 1,000,000, the FE was "healthy" because the current count was 0. It will only start OOM-ing again once the cluster performs enough new load jobs to fill that 1,000,000-object bucket again. Recommendation: Do not keep the values at 1,000,000. For an 8Gi-16Gi FE, a

label_keep_max_num

of 20,000 to 50,000 is typically the safe upper limit. References • sql-reference/sql-statements/cluster-management/nodes_processes/SHOW_PROC.md • administration/management/FE_configuration.md

Rumit Jain

04/17/2026, 4:56 PM

@Rocky How to read these html's inside

proc_profile/

? And I see lot of files in this directory. Is it good sign?

Rocky

04/17/2026, 4:56 PM

The files in the

proc_profile/

directory are Flame Graphs generated by the StarRocks FE's internal self-profiling daemon. Since StarRocks v3.2, this feature is enabled by default to help administrators diagnose performance issues and memory leaks without requiring manual intervention. 1. How to Read These HTML Files You do not need to download and open these files manually (though you can if you want). The intended way to read them is through the StarRocks Web UI: 1. Open the Web UI: Go to

http://<FE_IP>:8030

(default credentials:

root

with no password, or your configured admin credentials). 2. Navigate to Proc Profiles: Click on the "proc profiles" tab in the top navigation bar. 3. Choose FE: Select the FE sub-tab. 4. View the Flame Graph: You will see a list of files corresponding to those in your directory. Click the "View" link next to a specific timestamp. * CPU Profile: Shows where the FE is spending its CPU time (e.g., query planning, metadata replay). * Mem Profile: Shows where memory is being allocated (not necessarily what is currently held, but what is "churning" memory). 2. Is it a "good sign" to see many files? Yes, it is perfectly normal. By default, StarRocks FE runs a background collector that samples CPU and memory usage periodically. * Default Behavior: Each collection run lasts for 120 seconds (

proc_profile_collect_time_s

). * Retention: The FE is configured to keep these files for 1 day (

proc_profile_file_retained_days

) or until the directory reaches 2GB (

proc_profile_file_retained_size_bytes

). Because it samples so frequently, you will naturally see dozens of files in that directory. It is not an indication of a crash; rather, it's the "black box recorder" of your database. 3. Using them to confirm your OOM Since you were experiencing OOMs, these files are now your best friend: * Find a
mem-profile
from just before a known crash time. * Look for wide bars at the bottom of the Flame Graph. If you see large sections labeled

com.starrocks.load.LoadMgr

com.starrocks.common.util.LabelKeepTask

, it confirms that the Label Retention (the

label_keep_max_num

we discussed) was indeed the memory hog. Pro-Tip: Reducing Disk Usage If these files are taking up too much space on your EKS persistent volumes, you can tune these settings in your

fe.conf

(or Helm

fe_configs

Copy code

bash
# Keep only 12 hours of profiles instead of 24
proc_profile_file_retained_days = 0.5
# Limit the directory to 1GB
proc_profile_file_retained_size_bytes = 1073741824

Summary * Read them via: Web UI port 8030 -> "proc profiles". * "Lots of files" status: Normal background activity. * Value: Extremely high for debugging the OOM you just had. Look for the

mem-profile

nearest to the crash. References • administration/management/FE_configuration.md • administration/management/proc_profile.md

Rumit Jain

04/20/2026, 4:19 AM

@Rocky FE pods are going down with OOM. Seeing these logs:

Copy code

[MemoryUsageTracker.trackMemory():165] (0ms) Module Load - LoadMgr estimated 65MB of memory. Contains LoadJob with 315839 object(s).

Rocky

04/20/2026, 4:20 AM

The log you are seeing is the "smoking gun" confirming the memory pressure, despite the "65MB" number looking small. Here is the breakdown of why this is causing your OOM and how to interpret it. 1. The Estimation is a "Shallow" Estimate The

MemoryUsageTracker

in StarRocks provides an estimation, not a precise byte-for-byte measurement of the JVM heap. * The Log:

Module Load - LoadMgr estimated 65MB ... Contains LoadJob with 315839 object(s).

* The Reality: 65MB for 315,839 objects averages to about 215 bytes per object. In Java, a

LoadJob

object is much larger than 215 bytes when you include its "Deep Size" (the Strings for labels, the maps of task statuses, the list of involved tablets, etc.). * The Impact: Even if each job actually consumes just 10KB of total heap (a very conservative estimate for a job with metadata), 315,000 jobs would occupy ~3.1 GB of your heap. On an 8Gi heap, that is nearly 40% of your total memory just for "history," which is enough to trigger aggressive GC and OOMs when combined with other metadata. 2. Why 315,839 is the critical number The default value for

label_keep_max_num

is 1,000. You have it set to 1,000,000. Even though you are only at 315,000 right now, your FE is already struggling because: 1. Object Overhead: Managing 315k active objects in the

LoadMgr

map creates massive overhead for the JVM Garbage Collector. 2. Shared-Data Metadata: Since you are in

shared_data

mode, the FE is also managing S3 tablet metadata and performing cluster snapshots every 10 minutes. This requires significant "working memory" that is being crowded out by the 315k load jobs. 3. How to verify the "Real" memory cost If you want to see the actual size these objects are taking, run this command inside the FE pod:

Copy code

bash
# This will show you the top memory-consuming classes
jmap -histo:live 1 | grep -E "LoadJob|StreamLoadTask|String" | head -n 20

You will likely see

java.lang.String

at the very top with several gigabytes of usage. Most of those strings are the Labels and Error Messages attached to those 315,839 load jobs. 4. Immediate Fixes You must bring the object count down to a level that fits in an 8Gi-16Gi container. 1. Lower the Limit significantly: Change your config to a more reasonable value. Even for high-frequency loading, 50,000 is usually more than enough.

Copy code

yaml
    label_keep_max_num = 50000
    label_keep_max_second = 86400  # 24 hours

2. Clean up the current bloat: Lowering the config and restarting should trigger a cleanup. If the FE crashes before it can clean up, you may need to temporarily increase the pod memory to 16Gi just to let it start, perform the cleanup, and then scale it back down. 3. Adjust Heap Ratio: Ensure your

-Xmx

is not equal to your container limit. For an 8Gi container, use

-Xmx6g

. For a 16Gi container, use

-Xmx12g

. The "missing" 2-4Gi is required by the JVM for overhead and by the OS to prevent K8s from killing the pod. Summary The 65MB estimate is misleading because it likely only counts the "shallow" size of the manager's internal array. The 315,839 objects is the true indicator of the problem. Reducing your label retention is the only way to stabilize the FE with your current memory resources. References • loading/loading_introduction/loading_considerations.md • administration/management/monitoring/metrics.md • best_practices/audit_log_resource_group.md • administration/management/resource_management/Memory_management.md

Rocky

04/20/2026, 4:20 AM

• administration/management/BE_configuration.md • best_practices/primarykey_table.md

Rumit Jain

04/20/2026, 4:23 AM

@Rocky I am seeing this error:

Copy code

starrocks@kube-starrocks-fe-1:/opt/starrocks$ jmap -histo:live 1 | grep -E "LoadJob|StreamLoadTask|String" | head -n 20
Exception in thread "main" com.sun.tools.attach.AttachNotSupportedException: Unable to open socket file /tmp/.java_pid1: target process 1 doesn't respond within 10500ms or HotSpot VM not loaded
	at jdk.attach/sun.tools.attach.VirtualMachineImpl.<init>(VirtualMachineImpl.java:104)
	at jdk.attach/sun.tools.attach.AttachProviderImpl.attachVirtualMachine(AttachProviderImpl.java:58)
	at jdk.attach/com.sun.tools.attach.VirtualMachine.attach(VirtualMachine.java:207)
	at jdk.jcmd/sun.tools.jmap.JMap.executeCommandForPid(JMap.java:128)
	at jdk.jcmd/sun.tools.jmap.JMap.histo(JMap.java:202)
	at jdk.jcmd/sun.tools.jmap.JMap.main(JMap.java:112)
starrocks@kube-starrocks-fe-1:/opt/starrocks$

Rocky

04/20/2026, 4:23 AM

The

AttachNotSupportedException

error occurs because the Java "Attach Mechanism" (which

jmap

uses) cannot establish a connection with the JVM. In a Kubernetes environment, this is usually due to one of three reasons: 1. User Mismatch (Most Common) The

jmap

command must be executed by the exact same user who is running the StarRocks FE process. * Check: Run

ps -ef | grep java

inside the pod. * Action: If the process is owned by

root

but you are logged in as

starrocks

(or vice versa), the attach will fail. * Solution: Use

sudo -u <user>

or switch users before running the command. 2. PID 1 is a Shell Script In many StarRocks Docker images, PID 1 is the

start_fe.sh

script, not the Java process itself. Java tools cannot "attach" to a bash script. * Action: Find the real PID of the Java process:

Copy code

bash
    ps -ef | grep java

* Solution: Use the actual PID (e.g.,

) instead of

in your command:

Copy code

bash
    jmap -histo:live <REAL_PID> | grep -E "LoadJob|StreamLoadTask|String" | head -n 20

3. The JVM is "Busy" (Long GC or Safepoint) If your FE is currently OOM-ing or experiencing "Stop-the-World" Garbage Collection, it cannot reach a "Safepoint" to respond to

jmap

. * The logic: To provide a

:live

histogram, the JVM must pause everything to count objects. If the JVM is already frozen trying to clear 315,000 LoadJobs from memory,

jmap

will timeout. * Verification: Check your

fe.gc.log

(usually in the

log/

directory). If you see constant

Full GC

entries, the FE is too busy to talk to

jmap

. ────────── How to get the data despite this error If the standard

jmap

fails, try these alternatives: Alternative A: Use
jcmd
(More robust)

jcmd

is the modern replacement for

jmap

and often works better in containers.

Copy code

bash
# Find real PID first, then:
jcmd <REAL_PID> GC.class_histogram | grep -E "LoadJob|StreamLoadTask|String" | head -n 20

Alternative B: Force Mode (The "Hammer") If the process is truly hung and won't respond, you can use the

-F

(Force) flag. Warning: This will freeze the FE completely while it runs.

Copy code

bash
jmap -F -histo <REAL_PID> | grep -E "LoadJob|StreamLoadTask|String" | head -n 20

Why this confirms the OOM The fact that you have 315,839 LoadJob objects while the FE is struggling is the confirmation. * Memory Math: In Java, a

LoadJob

is a complex object. It doesn't just hold a "label" string; it holds the job's history, tablet distribution, error URLs, and status metadata. * The "Deep" Size: While the

MemoryUsageTracker

estimates 65MB (shallow size), the "Deep Heap" (the memory actually occupied) for 315k jobs is likely 3GB to 5GB. * Conclusion: On an 8Gi container (which usually has a ~6Gi JVM Heap), having 4GB+ occupied just by "job history" leaves no room for actual queries or metadata management, leading to the OOM. Recommendation: You should definitely lower

label_keep_max_num

to 50,000 or less. There is no performance benefit to keeping 300k+ historical labels in memory. References • administration/Meta_recovery.md

Rumit Jain

04/20/2026, 4:24 AM

@Rocky What should be the right value of this if my fe pods in lower envs are on 8Gib Memory and 3CPU:

Copy code

JAVA_OPTS="-Dlog4j2.formatMsgNoLookups=true -Xmx8192m -XX:+UseG1GC -Xlog:gc*:${LOG_DIR}/fe.gc.log.$DATE:time -XX:ErrorFile=${LOG_DIR}/hs_err_pid%p.log -Djava.security.policy=${STARROCKS_HOME}/conf/udf_security.policy"

Rocky

04/20/2026, 4:25 AM

For an 8GiB Kubernetes pod, your current

JAVA_OPTS

are the direct cause of the OOM crashes. Setting

-Xmx8192m

on an 8GiB pod is a "guaranteed crash" scenario because the JVM heap is not the only thing using memory. 1. The Core Problem: Heap vs. Pod Limit When you set

-Xmx8192m

, the JVM thinks it has 8GiB for the Heap only. However, the Pod's 8GiB limit also has to cover: * Metaspace: (Stores class metadata) ~200MB-500MB. * Code Cache: (JIT compiled code) ~128MB-256MB. * Thread Stacks: Each thread uses ~1MB. StarRocks FE can have hundreds of threads. * Direct Memory: Used for network buffers and certain metadata operations. * OS Overhead: The container's base processes and kernel memory. Result: The JVM will try to use all 8GiB for the heap, but once the total usage hits 8GiB (likely when the heap is only at 7GiB), the Kubernetes OOMKiller will instantly kill the pod. The JVM won't even have time to throw an

OutOfMemoryError

or log a GC event. 2. Recommended
JAVA_OPTS
for 8GiB Pod You should follow the 75% Rule for containerized Java applications. For an 8GiB pod, your heap should be 6GiB. Update your configuration to this:

Copy code

bash
JAVA_OPTS="-Dlog4j2.formatMsgNoLookups=true -Xms6144m -Xmx6144m -XX:+UseG1GC -Xlog:gc*:${LOG_DIR}/fe.gc.log.$DATE:time -XX:ErrorFile=${LOG_DIR}/hs_err_pid%p.log -Djava.security.policy=${STARROCKS_HOME}/conf/udf_security.policy"

Key Changes: * -Xmx6144m
(6GiB): This leaves ~2GiB of "headroom" for the Metaspace, stacks, and OS. This prevents the K8s OOMKiller from intervening. * `-Xms6144m`: Setting the initial heap (

Xms

) equal to the max heap (

Xmx

) is a StarRocks best practice. It prevents performance dips caused by the JVM constantly resizing the heap. 3. Why 8GiB is "Dangerously Small" for you As we saw in your logs, your FE is tracking 315,839 load jobs. * Official StarRocks hardware recommendations suggest a minimum of 16GiB for the FE. * With 300k+ objects, your "Metadata" is likely consuming 3GB-4GB of that 6GB heap already. * This leaves very little room for query planning or temporary objects, which is why you are seeing frequent GC and high CPU. Final Recommendation 1. Apply the 6GiB limit above immediately to stop the Pod from being killed by Kubernetes. 2. Lower
label_keep_max_num
to

fe.conf

. This is the most effective way to reduce the actual memory pressure so the FE can survive on 8GiB. 3. If possible, increase the Pod limit to 16GiB and set

-Xmx12g

. This is the "safe zone" for a production-like lower environment. References • faq/fe_mem_faq.md • faq/Others.md • administration/management/FE_configuration.md • quick_start/helm.md • faq/operation_maintenance_faq.md

Rumit Jain

04/20/2026, 4:35 AM

@Rocky What is this log line?

Copy code

2026-04-20 04:34:45.849Z INFO (starrocks-mysql-nio-pool-24|256) [JournalObservable.waitOn():49] waiting for the observer to replay journal to 3728748 with timeout: 14400000 ms

Rocky

04/20/2026, 4:35 AM

This log line is a byproduct of the Metadata Synchronization mechanism between your StarRocks FE nodes. What is happening? In a StarRocks cluster with multiple FEs (Leader, Followers, and Observers), only the Leader FE is allowed to write metadata (like starting a load job or creating a table). 1. Request Forwarding: You likely sent a request (like a

SELECT

INSERT

, or a status check) to a Non-Leader FE (Follower or Observer). 2. Leader Execution: The Non-Leader FE forwarded the request to the Leader. The Leader executed it and assigned it a "Journal ID" (in your case,

). 3. Consistency Check: To ensure you don't see "stale" data, the Non-Leader FE must wait until its own local metadata has "caught up" to that Journal ID before it responds to you. 4. The Log: This line means the FE thread handling your connection is now "parking" (waiting) for the internal Replayer thread to finish processing all journals up to

. Why are you seeing this now? This log is perfectly normal in small amounts, but seeing it frequently—especially alongside your OOM issues—is a sign of Metadata Congestion: * The Replayer is blocked by GC: Because your FE is under extreme memory pressure (the 315,000 load jobs we discussed), the Java Garbage Collector is likely performing frequent "Stop-the-World" pauses. This stops the Replayer thread from doing its job, causing other threads to wait indefinitely. * Journal Bloat: Every one of those 315,839 load jobs creates multiple journal entries (Start, Commit, Finish). The Replayer has to process every single one of these. If you are doing massive amounts of small loads, the Replayer can't keep up with the Leader. * The Timeout: The

14400000 ms

(4 hours) is the default safety timeout. If the Replayer doesn't catch up by then, the client request will eventually fail with a "Metadata not in sync" error. Connection to your OOM This log confirms that your FE is "stalling." The memory pressure from the 315k load jobs is preventing the FE from keeping its metadata in sync. To resolve this: 1. Clean up the labels: As suggested previously, reduce

label_keep_max_num

. This will reduce the number of journals the Replayer has to track and replay. 2. Check Replay Progress: You can see how far behind this FE is by running:

Copy code

sql
    SHOW FRONTENDS;

Look at the

ReplayedJournalId

column compared to the Leader's

LastHeartbeat

. If the gap is large (thousands of IDs), the FE is struggling to keep up. 3. Adjust Memory: Ensure the FE has enough Heap (the 6GiB vs 8GiB pod limit discussed earlier) so the Replayer thread isn't constantly interrupted by GC. References • knowledge/trouble_shooting/fe_deadlock_problems.md

Rumit Jain

04/20/2026, 4:36 AM

@Rocky How to check this? Adjust Memory: Ensure the FE has enough Heap (the 6GiB vs 8GiB pod limit discussed earlier) so the Replayer thread isn't constantly interrupted by GC.

Rocky

04/20/2026, 4:36 AM

To verify if your FE is suffering from insufficient heap and GC-induced stalls, you should check three specific areas: the real-time JVM stats, the GC logs, and the metadata replay status. 1. Check Real-Time Heap Usage (
jstat
) This is the most reliable way to see if the Old Generation (where long-lived metadata like

LoadJob

lives) is full. 1. Find the real Java PID (not

ps -ef | grep java

2. Run `jstat`:

Copy code

bash
   # jstat -gcutil <PID> 1000 5
   jstat -gcutil 25 1000 5

* What to look for: Look at the O
(Old Generation) column. If it is consistently 90%
, your heap is too small for the amount of metadata you are carrying. * GC Count: Look at

FGC

(Full GC count). If this number is increasing every few seconds, the FE is "thrashing"—spending all its CPU time trying to clear memory and failing. 2. Check the GC Logs for "Stalls" Since you have

-Xlog:gc*

enabled in your

JAVA_OPTS

, you can check the

fe.gc.log

directly.

Copy code

bash
grep -E "Pause Full|Pause Young" /opt/starrocks/log/fe.gc.log | tail -n 20

* The "Replayer Thread" connection: The Replayer thread (which handles the

waitOn():49

log you saw) is a JVM thread. When a

Full GC

happens, it triggers a "Stop-the-World" pause. * The Evidence: If you see pauses like

Pause Full (G1 Evacuation Pause) 5000ms

, it means the entire FE stopped for 5 seconds. During those 5 seconds, the Replayer could not process journals, which is exactly why you see the

waiting for the observer to replay

logs. 3. Check Replay Lag (
SHOW FRONTENDS
) This confirms if the metadata "wait" is actually resulting in a lag. Run this in your MySQL client:

Copy code

sql
SHOW FRONTENDS\G

* Compare
ReplayedJournalId
vs `LastHeartbeat`: On the Follower/Observer nodes, look at the

ReplayedJournalId

. If it is significantly lower (by hundreds or thousands) than the Leader's ID, the node is falling behind because the CPU/Memory is too busy with GC to replay the journals. 4. Check the Memory Usage Tracker If you are on StarRocks v3.3.7+, look in your

fe.log

for a periodic report from the

MemoryUsageTracker

Copy code

bash
grep "MemoryUsageTracker" /opt/starrocks/log/fe.log | tail -n 5

It will break down which modules are using memory. You will likely see LoadManager
or EditLog
consuming several gigabytes. ────────── Summary Checklist Symptom | Meaning

jstat

Old Gen > 90% | Your 6GB heap is physically full of metadata.

FGC

count increasing | FE is in a "death spiral" of Garbage Collection.

SHOW FRONTENDS

lag | Replayer thread is stalled (likely due to the GC above).

fe.log

wait journals | Confirmation that query threads are blocked by the Replayer stall. Action: If these checks confirm the heap is full, you must reduce the metadata footprint by lowering

label_keep_max_num

(e.g., to

) and restarting the FE to clear the bloated history. References • faq/fe_mem_faq.md • knowledge/trouble_shooting/fe_memory_problems.md • administration/management/logs.md • administration/Meta_recovery.md • administration/management/resource_management/Memory_management.md • developers/jemalloc_heap_profile.md

Rumit Jain

04/20/2026, 4:37 AM

@Rocky This is this putput:

Copy code

jstat -gcutil 18 1000 5
  S0     S1     E      O      M     CCS    YGC     YGCT     FGC    FGCT     CGC    CGCT       GCT
  0.00 100.00   8.90  86.30  99.43  97.39    137     9.043     0     0.000    38     0.249     9.293
  0.00 100.00  26.99  86.30  99.43  97.39    137     9.043     0     0.000    38     0.249     9.293
  0.00  95.00  25.63  86.37  99.41  97.40    138     9.082     0     0.000    38     0.249     9.331
  0.00 100.00  24.18  86.37  99.42  97.40    139     9.119     0     0.000    38     0.249     9.368
  0.00  97.73  14.53  86.55  99.43  97.40    140     9.176     0     0.000    38     0.249     9.426

Rocky

04/20/2026, 4:38 AM

Based on your

jstat

output, your FE is in a highly unstable state. Here is the breakdown of why your pod is likely crashing or stalling: 1. Metaspace & CCS are Exhausted (99%+) * The Data:

(Metaspace) is at 99.43% and

CCS

is at 97.40%. * The Risk: These areas store class metadata. When they hit ~100%, the JVM triggers a "Metadata GC threshold" which forces the entire FE to stop everything (Stop-The-World) to try and clean up classes. * The Container Trap: Since your Pod is limited to 8GiB and you are likely using a large heap, there is almost no physical memory left for the Metaspace to expand. If it tries to grow even slightly, the Kubernetes OOMKiller will kill the pod instantly. 2. Old Generation is Crowded (86.55%) * The Data:

is at 86.55%. * The Meaning: In the G1GC algorithm (which you are using), hitting >80% in the Old Gen triggers constant background "Marking" cycles (

CGC

in your output is 38). This consumes massive amounts of CPU and slows down the Replayer thread. * The Cause: This 86% is where those 315,000 load jobs are living. They are "sticky" objects that won't go away until they expire. 3. High GC Frequency * The Data:

YGC

(Young GC) count went from 137 to 140 in just 5 seconds (1 GC every 1.6 seconds). * The Impact: Every time a GC happens, your FE pauses. If you have 3 GCs in 5 seconds, your FE is spending a significant portion of its life paused. This is exactly why you see the log:

waiting for the observer to replay journal...

. The Replayer thread is being paused every second by the GC. ────────── Immediate Action Plan To stabilize this FE, you need to reduce the pressure on both the Heap and the Metaspace: Step 1: Reduce Metadata Footprint Modify your

fe/conf/fe.conf

to drastically lower the history retention. This will clear out the Old Gen (the 86.55%):

Copy code

properties
# Default is often 3-7 days or 200,000+ records. 
# Lowering this will force FE to delete old load metadata.
label_keep_max_num = 20000
label_keep_max_second = 14400

Step 2: Fix
JAVA_OPTS
(The 75% Rule) If your Pod is 8GiB, do not use

-Xmx8192m

. You must leave room for the 99% full Metaspace and Thread Stacks. Update to:

Copy code

bash
JAVA_OPTS="-Xms6144m -Xmx6144m -XX:MaxMetaspaceSize=512m -XX:+UseG1GC ..."

* 6144m (6GiB): Gives the JVM heap room to breathe without hitting the 8GiB Pod limit. * MaxMetaspaceSize=512m: Puts a hard cap on Metaspace so it doesn't expand and trigger a K8s OOM kill. Step 3: Restart and Monitor After the restart: 1. Run

SHOW FRONTENDS;

to ensure

ReplayedJournalId

is catching up to the Leader. 2. Run

jstat -gcutil <pid> 1000 5

again. You want to see

(Old Gen) drop below 60% and

(Metaspace) stabilize with some headroom. Recommendation: If this is a production or critical environment, an 8GiB Pod is too small for StarRocks FE. We strongly recommend increasing the Pod limit to 16GiB and setting

-Xmx12g

. References • administration/Meta_recovery.md • knowledge/trouble_shooting/fe_memory_problems.md • faq/fe_mem_faq.md

Rumit Jain

04/20/2026, 5:51 AM

@Rocky Reduced the

Copy code

label_keep_max_num=50000
label_keep_max_second=86400
JAVA_OPTS="-Dlog4j2.formatMsgNoLookups=true -Xms6144m -Xmx6144m -XX:+UseG1GC -Xlog:gc*:${LOG_DIR}/fe.gc.log.$DATE:time -XX:ErrorFile=${LOG_DIR}/hs_err_pid%p.log -Djava.security.policy=${STARROCKS_HOME}/conf/udf_security.policy"

And then ran this

Copy code

ps -ef | grep java
starroc+      18       1 16 05:28 ?        00:03:28 /lib/jvm/java-17-openjdk/bin/java -Dlog4j2.formatMsgNoLookups=true -Xms6144m -Xmx6144m -XX:+UseG1GC -Xlog:gc*:/opt/starrocks/fe/log/fe.gc.log.:time -XX:ErrorFile=/opt/starrocks/fe/log/hs_err_pid%p.log -Djava.security.policy=/opt/starrocks/fe/conf/udf_security.policy com.starrocks.StarRocksFE --host_type FQDN
starroc+     522     519  0 05:49 pts/0    00:00:00 grep java
starrocks@kube-starrocks-fe-1:/opt/starrocks$ jstat -gcutil 18 1000 5
  S0     S1     E      O      M     CCS    YGC     YGCT     FGC    FGCT     CGC    CGCT       GCT
  0.00  96.15  33.09  54.93  99.23  96.74     29     1.354     0     0.000     8     0.012     1.366
  0.00  96.15  33.30  54.93  99.23  96.74     29     1.354     0     0.000     8     0.012     1.366
  0.00  96.15  33.51  54.93  99.23  96.74     29     1.354     0     0.000     8     0.012     1.366
  0.00  96.15  33.61  54.93  99.23  96.74     29     1.354     0     0.000     8     0.012     1.366
  0.00  96.15  33.82  54.93  99.23  96.74     29     1.354     0     0.000     8     0.012     1.366

Rocky

04/20/2026, 5:51 AM

This is a significant improvement! Your Old Generation (O) has dropped from 86% to 54%, which confirms that reducing the

label_keep_max_num

successfully purged the old load metadata that was choking your FE. However, your

jstat

reveals a remaining bottleneck that is likely still causing the

waiting for journal

logs and general slowness: The Problem: Metaspace Exhaustion * M (Metaspace): 99.23% * CCS (Compressed Class Space): 96.74% Metaspace is where the JVM stores class definitions. Because it is nearly 100% full, the JVM is likely triggering Metadata GCs. These are "Stop-the-World" pauses just like a Full GC. This is why you are still seeing the

waiting for the observer to replay journal

logs—the thread responsible for replaying journals is being paused because the JVM is desperately trying to find room for new classes (often generated during SQL planning or UDF execution). The Danger of the 8GiB Pod Limit You have set your Heap to 6GiB (

-Xmx6144m

) inside an 8GiB Pod. This is very tight: 1. Heap: 6 GiB 2. Metaspace: ~512 MiB (currently full) 3. Thread Stacks: (e.g., 200 threads * 1MB) = 200 MiB 4. Off-heap/Netty buffers: 200-500 MiB Total: ~7.2 GiB. If Metaspace tries to expand to 100% (or if a heavy query uses extra off-heap memory), your total usage will hit 8GiB, and the Kubernetes OOMKiller will kill the pod instantly without writing a log. ────────── Recommended Next Steps 1. Give Metaspace Room to Breathe Update your

JAVA_OPTS

to explicitly define Metaspace limits. This prevents the JVM from "thrashing" at the 99% mark:

Copy code

bash
# Add these to your existing JAVA_OPTS
-XX:MetaspaceSize=512m -XX:MaxMetaspaceSize=512m

Note: If the FE fails to start or crashes with "Out of Metadata Space," it means StarRocks truly needs more than 512MB for classes, and you must increase the Pod memory. 2. Verify the "Journal" Log Check your

fe.log

again. Do you still see the

waiting for the observer to replay journal

lines? * If YES: The Metaspace at 99% is still causing pauses. * If NO: Your stability has returned, but you are still at risk of an OOM crash. 3. Long-term: Increase Pod Resources The official StarRocks recommendation for an environment with up to 1 million tablets is 16GiB of RAM for the FE. * An 8GiB pod for a StarRocks FE is considered "minimal" and prone to the issues you are seeing. * If you can increase the Pod limit to 12GiB or 16GiB, you can set

-Xmx8g

and

-XX:MaxMetaspaceSize=1g

, which would make the cluster much more resilient to load spikes. Summary of current status: * Heap: Healthy (54% usage). * Metaspace: Critical (99% usage - causing stalls). * Stability: Unstable (Risk of K8s OOM kill). Action: Apply the Metaspace JVM flags and, if possible, increase the Pod memory to 12GiB+. References • faq/Others.md • knowledge/trouble_shooting/fe_memory_problems.md • administration/management/FE_configuration.md • administration/Meta_recovery.md • faq/fe_mem_faq.md • administration/management/monitoring/Monitor_and_Alert.md

Rumit Jain

04/20/2026, 7:49 AM

@Rocky what kind of matrix i can have for this in my grafana and send alert

Rocky

04/20/2026, 7:50 AM

To effectively monitor and alert on the FE memory and replay issues you've been experiencing, you should focus on these specific Prometheus metrics and Grafana alerts. 1. The "Replay Stall" Alert (Critical) When the Replayer thread is blocked by GC, the "Meta Log Count" (the gap between logs written and logs checkpointed/replayed) will spike. * Metric:

starrocks_fe_meta_log_count

* Grafana Panel: Gauge or Time Series. * Alert Rule:

starrocks_fe_meta_log_count > 100000

* Meaning: If this is high, the FE is failing to process its metadata journals (checkpoints are failing). This is the "can't keep up" signal you saw in your logs. 2. The "Metaspace" Alert (Critical for your current state) Your Metaspace is at 99%. You need to monitor when the JVM non-heap memory is exhausted. * Metric:

jvm_non_heap_used_bytes

jvm_non_heap_committed_bytes

* Grafana Panel: Time Series (Ratio). * Alert Rule:

(jvm_non_heap_used_bytes / jvm_non_heap_committed_bytes) > 0.95

* Meaning: If Metaspace hits 100%, the JVM will trigger frequent Stop-The-World GCs or the Pod will be OOM-Killed by Kubernetes. 3. FE JVM Heap Usage Alert This monitors the Old Generation pressure caused by having too many load jobs (

label_keep_max_num

). * Metric:

jvm_heap_used_bytes

jvm_heap_size_bytes

* Grafana Panel: Time Series (Percentage). * Alert Rule:

(jvm_heap_used_bytes / jvm_heap_size_bytes) > 0.80

* Meaning: When Heap exceeds 80%, G1GC begins aggressive background marking, which steals CPU from the metadata replayer. 4. GC Pause Duration Alert This tells you if your FE is "stuttering" due to memory pressure. * Metric:

jvm_gc_collection_seconds_sum

(increase over time) * PromQL Alert:

rate(jvm_gc_collection_seconds_sum[1m]) > 0.2

* Meaning: This alerts if the FE is spending more than 20% of its time (12 seconds out of every minute) doing nothing but Garbage Collection. 5. Follower/Observer Replay Lag (Visual Check) If you have multiple FEs, you should monitor the gap between the Leader's journal ID and the Follower's replayed ID. * Metrics: *

starrocks_fe_max_journal_id

(The current ID on the Leader) *

starrocks_fe_edit_log_replayed_id

(The ID reached by this node) * Alert Rule:

starrocks_fe_max_journal_id - starrocks_fe_edit_log_replayed_id > 1000

* Meaning: The Follower node is falling behind the Leader's metadata state. Queries on this Follower will be stale or will wait/timeout. ────────── Suggested Grafana Dashboard Layout Panel | Type | Metric/Formula FE Metadata Health | Gauge |

starrocks_fe_meta_log_count

JVM Heap Usage | % Gauge |

jvm_heap_used_bytes / jvm_heap_size_bytes

Metaspace Usage | % Gauge |

jvm_non_heap_used_bytes / jvm_non_heap_max_bytes

GC Stop-the-world | Time Series |

irate(jvm_gc_collection_seconds_sum[1m])

Edit Log Write Latency | Time Series |

starrocks_fe_edit_log_write_latency_ms

Pro Tip: If you use the StarRocks Grafana Dashboard template (available on GitHub or StarRocks docs), these are usually included in the "FE" row. Look specifically for the "FE JVM" and "FE Meta" sections. References • administration/management/monitoring/Monitor_and_Alert.md • administration/management/monitoring/metrics.md • administration/management/FE_configuration.md • faq/fe_mem_faq.md • administration/Meta_recovery.md • knowledge/trouble_shooting/fe_memory_problems.md References

Rocky

04/20/2026, 7:50 AM

• administration/management/monitoring/metrics.md • administration/management/monitoring/Monitor_and_Alert.md

Rumit Jain

06/21/2026, 3:35 AM

@Rocky What should be the runbook for CN pods High Disk utilization alert of more than 90%. What is the starrocks recommendation on thie alert and metrics?

Rocky

06/21/2026, 3:35 AM

For CN (Compute Node) pods, which are primarily used in shared-data (Cloud Native) environments, high disk utilization (>90%) is usually driven by Data Cache, Query Spilling, or Logs/Core Dumps rather than persistent data storage. Runbook: CN Pod High Disk Utilization (>90%) Step 1: Identify the Culprit Exec into the pod and run:

Copy code

bash
du -sh /opt/starrocks/cn/* --max-depth=1

* storage/
(specifically
storage/star_cache
): This is the Data Cache. It is designed to fill up but should stay within its configured limit. * storage/spill
: This is intermediate data from heavy queries (spilling). If this is huge, a query is "exploding." * log/
: Check if

<http://cn.INFO|cn.INFO>

cn.WARNING

logs have grown massive due to error loops. * Core Dumps: Check for files named

core.<pid>

in the StarRocks home directory. ────────── Step 2: Immediate Mitigation Actions A. If caused by Data Cache: The cache is designed to be persistent and only evicts when it hits its limit. If it's pushing the disk to 90%, you likely over-allocated the cache size relative to the PV size. * Immediate fix: You can reduce the cache limit dynamically:

Copy code

bash
    curl http://<cn_ip>:<cn_http_port>/api/update_config?starlet_star_cache_disk_size_percent=70

* Clear cache (Nuclear option): Delete the contents of the

star_cache

directory and restart the pod. B. If caused by Query Spilling: Spilling occurs when a query's memory exceeds the limit and it writes intermediate results to disk. * Check active queries: Run

SHOW PROC '/current_queries'

. Look for queries with long durations. * Kill the heavy query:

KILL QUERY <query_id>;

* Clean up: Spilled files should be deleted automatically after the query ends, but if a CN crashed, you might need to manually delete files in

${storage_root_path}/spill/

. C. If caused by Logs or Core Dumps: * Logs: Delete old

.gz

logs in the

/log

directory. * Core Dumps: Delete

core.*

files. Important: Capture a stack trace before deleting if you need to debug a crash. ────────── StarRocks Recommendations & Metrics StarRocks recommends maintaining disk utilization below 80% to avoid performance degradation (I/O wait) and "Disk Full" errors during critical operations like metadata updates. 1. Key Metrics to Monitor Metric | Description | Alert Threshold

starrocks_be_disks_avail_capacity

| Percentage of available disk space. | Alert if

< 15%

starrocks_be_data_cache_disk_size

| Actual bytes used by the Data Cache. | Watch for parity with PV size

starrocks_be_query_spill_bytes

| Total bytes currently spilled to disk. | Alert if

> [50% of disk]

starrocks_be_disks_data_used_capacity

| Total capacity used on the storage path. | Alert if

> 90%

2. Prometheus Alert Recommendation Data Disk Capacity Alert:

Copy code

prometheus
(SUM(starrocks_be_disks_total_capacity) by (host, path) - SUM(starrocks_be_disks_avail_capacity) by (host, path)) 
/ SUM(starrocks_be_disks_total_capacity) by (host, path) * 100 > 90

────────── Preventative Configuration (CN Best Practices) To prevent this alert from recurring, update your

cn.conf

or Helm Chart: 1. Limit Data Cache: Ensure

starlet_star_cache_disk_size_percent

(default 80%) leaves enough room for logs and spilling. If your PV is small, set this to

. 2. Spill Storage Limit: You can limit the space used for spilling:

Copy code

properties
    # In cn.conf
    spill_storage_limit_gb = 20

3. Log Rotation: Ensure

sys_log_history_days

is set to a reasonable value (e.g.,

) to prevent log buildup. 4. PV Sizing: For CN nodes using Data Cache, the Persistent Volume should be sized at least 20-30% larger than the intended cache size. References • administration/management/monitoring/metrics.md

Rocky

06/21/2026, 3:35 AM

• administration/management/monitoring/alert.md • using_starrocks/caching/block_cache.md • administration/management/monitoring/Monitor_and_Alert.md References • administration/management/monitoring/Monitor_and_Alert.md • administration/management/monitoring/alert.md

Rumit Jain

06/21/2026, 3:45 AM

@Rocky Why my log dis is 0 even though I have mountelog pvc:

Copy code

starrocks@kube-starrocks-cn-0:/opt/starrocks$ du -sh /opt/starrocks/cn/* 2>/dev/null | sort -hr
2.3G	/opt/starrocks/cn/lib
192M	/opt/starrocks/cn/storage0
188M	/opt/starrocks/cn/storage1
17M	/opt/starrocks/cn/datadog
680K	/opt/starrocks/cn/www
224K	/opt/starrocks/cn/bin
28K	/opt/starrocks/cn/conf
0	/opt/starrocks/cn/storage
0	/opt/starrocks/cn/spill
0	/opt/starrocks/cn/log

I have this config as well in my values.yaml:

Copy code

cnEnvVars:
    - name: LOG_CONSOLE
      value: "1"

Rocky

06/21/2026, 3:45 AM

Your log directory is empty specifically because you have enabled Console Logging. 1. Why the
log
directory is empty In StarRocks, when you set the environment variable LOG_CONSOLE: "1"
, the internal logging framework (glog/log4j) redirects all output to stdout and stderr (the container's console). * Behavior: StarRocks stops writing to files like

<http://cn.INFO|cn.INFO>

cn.WARNING

, or

cn.out

in the

/opt/starrocks/cn/log

directory. * Result: The directory remains empty (0 bytes) even if you have a PVC mounted to it. * How to view logs: You should use

kubectl logs <pod_name>

to see your logs. 2. Why
storage
and
spill
are also 0 * spill/
: This directory is only used for temporary data when a query exceeds its memory limit and "spills" to disk. If you aren't running heavy queries that trigger spilling, this will stay empty. * storage/
: For a CN (Compute Node), this is primarily used for the Data Cache (Block Cache). If the cluster is new or you haven't queried any data from your data lake (S3/HDFS) yet, the cache hasn't been populated. How to fix it (if you want log files on the PVC) If you want the logs to be written as files to your PVC (for example, to keep a long history or use a sidecar for log collection), you must disable console logging in your `values.yaml`:

Copy code

yaml
cnEnvVars:
  - name: LOG_CONSOLE
    value: "0"  # Set to "0" or remove this variable entirely

Recommendation * In Kubernetes: It is actually standard practice to keep

LOG_CONSOLE: "1"

so that Kubernetes native log collectors (like Fluentd, Promtail, or Datadog) can pick up the logs directly from the container runtime. * Troubleshooting: If you need to see the logs, use:

Copy code

bash
    kubectl logs -f kube-starrocks-cn-0

* PVC Usage: If you keep

LOG_CONSOLE: "1"

, you can reclaim that disk space by removing the log PVC mount for the CN, as it isn't being used. References • administration/management/BE_configuration.md • administration/management/logs.md • administration/management/monitoring/Monitor_and_Alert.md • administration/management/FE_configuration.md • using_starrocks/caching/block_cache.md References • administration/management/BE_configuration.md • administration/management/FE_configuration.md • administration/management/logs.md

Rumit Jain

06/21/2026, 3:47 AM

@Rocky I have

Copy code

192M	/opt/starrocks/cn/storage0
188M	/opt/starrocks/cn/storage1

Then why this is coming as 0?

Copy code

0	/opt/starrocks/cn/storage

Rocky

06/21/2026, 3:47 AM

The reason

/opt/starrocks/cn/storage

is 0 bytes while

storage0

and

storage1

have data is due to how the StarRocks Kubernetes Operator (or Helm Chart) handles multiple storage volumes. 1. Multi-Disk Configuration In your

cn.conf

(or the equivalent configuration injected by Kubernetes), the

storage_root_path

is likely configured to use the two specific mount points:

Copy code

properties
storage_root_path = /opt/starrocks/cn/storage0;/opt/starrocks/cn/storage1

When StarRocks sees multiple paths defined, it writes data (like the Data Cache) directly into those specific directories. It does not use the default

/opt/starrocks/cn/storage

directory at all. 2. Why the "storage" directory exists The directory

/opt/starrocks/cn/storage

is the default internal path for StarRocks. It usually exists in the container image as a placeholder. Since your configuration explicitly points to

storage0

and

storage1

, the default folder remains untouched and empty (0 bytes). 3. Verification You can confirm this by checking the actual configuration inside the pod:

Copy code

bash
grep "storage_root_path" /opt/starrocks/cn/conf/cn.conf

You will likely see that it lists

storage0

and

storage1

separated by a semicolon. Summary for your Runbook * Ignore
/opt/starrocks/cn/storage
: It is a legacy/default placeholder that isn't being used. * Monitor
storage0
and
storage1
: These are your actual data disks (likely PVCs). If these hit 90%, that is when your alert will trigger. * Disk Skew: It is normal for

storage0

and

storage1

to have slightly different sizes as StarRocks distributes the cache/metadata across the available paths provided. References • administration/management/BE_configuration.md • deployment/shared_data/hdfs.md • deployment/shared_data/s3.md • deployment/shared_data/minio.md • deployment/shared_data/azure.md • deployment/shared_data/gcs.md • faq/Others.md • quick_start/shared-data.md

Rumit Jain

06/21/2026, 3:49 AM

@Rocky What are the advantages for having multiple "storage" directory?

Rocky

06/21/2026, 3:49 AM

Having multiple storage directories (e.g.,

storage0

storage1

) for your StarRocks CN (Compute Node) or BE (Backend) provides several technical and operational advantages: 1. Increased I/O Throughput and IOPS This is the primary advantage. StarRocks performs I/O operations in parallel across all configured storage paths. * Parallelism: When reading or writing Data Cache, StarRocks can utilize multiple threads to access different physical disks simultaneously. * Reduced Bottlenecks: If you have multiple SSDs/NVMe drives, using them as separate paths allows you to saturate the hardware performance better than a single large volume might. 2. Bypassing Cloud Provider Limits In Kubernetes environments (AWS, GCP, Azure), cloud providers often impose performance caps (IOPS and Throughput) per Persistent Volume (PV). * Example: If a single 500GB volume is capped at 3,000 IOPS, mounting two 500GB volumes (

storage0

and

storage1

) can effectively give your CN node 6,000 IOPS in total. * This is often more cost-effective than provisioning a single ultra-high-performance disk. 3. Capacity Aggregation It allows you to pool the storage capacity of multiple disks without needing a complex software RAID setup or Logical Volume Manager (LVM) at the OS level. StarRocks handles the distribution of data across the semicolon-separated paths in

storage_root_path

. 4. Better Handling of Query Spilling When queries exceed memory limits, they "spill" intermediate data to disk. * By configuring

query_scratch_dirs

(or using multiple storage paths), StarRocks can distribute the heavy write/read load of spilled data across multiple disks, preventing a single disk from becoming a bottleneck and slowing down the entire query. 5. Disk Load Balancing StarRocks monitors the usage of each path. Using the parameter

storage_high_usage_disk_protect_ratio

(default 0.1), the system intelligently avoids writing more data to a disk that is significantly fuller than the average of the others. This ensures your storage fills up evenly. 6. Fault Tolerance (Ignore Broken Disks) You can configure StarRocks to be resilient to partial disk failures. * By setting

ignore_broken_disk = true

, if one of your disks (e.g.,

storage1

) fails or becomes unmountable, the CN/BE can still start up and function using the remaining healthy disks (

storage0

), rather than crashing the entire pod. Summary for CN Nodes Specifically for CN pods, these directories are used for the Data Cache. Having multiple directories means your cache is larger, faster to read, and more resilient to the performance throttling of any single Kubernetes PV. References • administration/management/BE_configuration.md • faq/Deploy_faq.md • administration/management/monitoring/Monitor_and_Alert.md References • faq/Deploy_faq.md • faq/operation_maintenance_faq.md • administration/management/BE_configuration.md • faq/Others.md • deployment/deploy_manually.md

Rumit Jain

06/21/2026, 3:55 AM

@Rocky How data gets cached in multiple PVC in this case?

Rocky

06/21/2026, 3:55 AM

When you configure multiple storage paths (like

storage0

and

storage1

) in a StarRocks CN node, the Block Cache (Data Cache) distributes data across them using a hashing and load-balancing mechanism. Here is how the data ends up being spread across your multiple PVCs: 1. Fixed-Size Block Splitting When StarRocks reads data from remote storage (S3, HDFS, etc.), it doesn't cache the whole file as one piece. Instead, it splits the file into small, fixed-size blocks (the default is often 1 MB). 2. Unique Cache Keys Every single block is assigned a globally unique Cache Key. This key is generated based on: *

hash(filename)

fileModificationTime

blockId

(the sequence number of the block within that file) 3. Distribution Mapping (Hashing) StarRocks uses the Cache Key to determine which directory (

storage0

storage1

) a specific block should be written to. * Mechanism: It typically applies a hash function to the key and performs a modulo operation based on the number of available storage paths. * Result: This ensures that blocks from the same file are spread across all available disks, which prevents any single PVC from becoming a hotspot. 4. Load Balancing & I/O Parallelism Because the blocks are distributed across multiple PVCs: * Parallel Reads: When you run a query that needs a 128 MB file, StarRocks can read blocks from

storage0

and

storage1

at the same time. This effectively doubles your I/O throughput if you have two PVCs. * Capacity Balancing: The system monitors the free space on each path. If one disk becomes significantly fuller than the others, StarRocks' internal logic (controlled by parameters like

storage_high_usage_disk_protect_ratio

) will steer new cache writes toward the disks with more remaining capacity. 5. Why you see slightly different sizes In your case,

storage0

(192M) and

storage1

(188M) are very close in size. This small discrepancy is normal and happens because: * The hashing isn't perfectly 50/50 for small amounts of data. * Some blocks might have been evicted from one disk but not the other based on the LRU (Least Recently Used) policy. Summary Your data is "striped" at a block level across your PVCs. This is the optimal way to use multiple disks in Kubernetes because it overcomes the IOPS limits of a single cloud volume and maximizes query performance. References • data_source/data_cache.md • using_starrocks/caching/block_cache.md • administration/management/BE_configuration.md • administration/management/monitoring/Monitor_and_Alert.md • faq/Deploy_faq.md • faq/Others.md • faq/operation_maintenance_faq.md • using_starrocks/caching/query_cache.md References • data_source/data_cache.md • using_starrocks/caching/block_cache.md • using_starrocks/caching/query_cache.md

Rumit Jain

06/21/2026, 3:58 AM

@Rocky What is this config used for enable_datacache_disk_auto_adjust

Rocky

06/21/2026, 3:58 AM

The configuration

enable_datacache_disk_auto_adjust

(renamed from

datacache_auto_adjust_enable

in v4.0+) is used to enable Automatic Scaling for the Data Cache disk capacity. When this is enabled, StarRocks dynamically manages how much of your physical disk space is used for caching data from remote storage (S3, HDFS, etc.). It helps prevent your PVCs from filling up completely while maximizing cache usage when space is available. How it works: The system monitors the disk usage at regular intervals (defined by

datacache_disk_adjust_interval_seconds

) and performs the following actions: * Automatic Eviction (Shrinking): If the total disk usage exceeds the high threshold (default

90%

, set by

datacache_disk_high_level

), StarRocks will automatically delete (evict) older cache data to bring the usage back down to a "safe" level. * Automatic Expansion: If the disk usage is low (default below

60%

, set by

datacache_disk_low_level

) and the cache is currently full, the system will allow the cache to grow and occupy more space. * Target Level: The system aims to keep the disk usage around the "safe" level (default

80%

, set by

datacache_disk_safe_level

). Why use it with multiple PVCs? In your setup with

storage0

and

storage1

, this feature is particularly useful because: 1. Safety: It prevents a single disk from hitting 100% and causing potential I/O errors or "Disk Full" states for the node. 2. No Manual Tuning: You don't have to manually set a hard limit (like

datacache_disk_size = 100G

) for every disk. The system will simply use whatever space is available on those PVCs up to the safe threshold. 3. Efficiency: If you increase the size of your PVCs in Kubernetes later, StarRocks will automatically detect the new space and expand the cache to fill it without requiring a config change. Summary of related parameters: Parameter | Default | Description

enable_datacache_disk_auto_adjust

true

| Enables/Disables this logic.

datacache_disk_high_level

| Percentage at which eviction starts.

datacache_disk_safe_level

| The target percentage for disk usage.

datacache_disk_low_level

| Percentage below which expansion is allowed. References • administration/management/BE_configuration.md • data_source/data_cache.md • sql-reference/System_variable.md • using_starrocks/caching/block_cache.md References • administration/management/BE_configuration.md • data_source/data_cache.md • sql-reference/System_variable.md

Rumit Jain

06/21/2026, 4:09 AM

@Rocky I have fe logs also printing in stdout in eks, but gc logs still getting saved in /opt/starrocks/fe/log. Also other logs are 0 in size. Where does CN stores its gc log?

Copy code

-rw-r--r--. 1 starrocks starrocks   111897 Jun 21 04:06 fe.gc.log.20260621-014927
-rw-r--r--. 1 starrocks starrocks        0 Mar 13 03:29 fe.internal.log
-rw-r--r--. 1 starrocks starrocks        0 Mar 13 03:29 fe.log
-rw-r--r--. 1 starrocks starrocks        0 Mar 13 03:29 fe.profile.log
-rw-r--r--. 1 starrocks starrocks        0 Mar 13 03:29 fe.warn.log

Rocky

06/21/2026, 4:09 AM

In a StarRocks EKS deployment where you have redirected logs to

stdout

, the behavior you are seeing is expected due to how different logging layers (Application vs. JVM) handle output. 1. Why FE GC logs are still on disk The standard FE logs (

fe.log

fe.warn.log

) are managed by Log4j, the application-level logging framework. When you enable redirection to console, Log4j sends these logs to

stdout

, which is why the local files are 0 bytes. However, GC logs (

fe.gc.log

) are generated by the JVM itself (at the native runtime level), not by Log4j. They are typically configured via a startup parameter like this:

Copy code

bash
-Xlog:gc*:/opt/starrocks/fe/log/fe.gc.log:time,tags:filecount=10,filesize=100M

Because the JVM is instructed to write directly to a specific file path, it ignores the application's Log4j configuration and continues writing to the local disk. 2. Where does CN store its GC log? By default, the CN (Compute Node) does not have a dedicated
cn.gc.log
file. The CN is primarily a C++ process. While it can launch an embedded JVM (via JNI) to handle certain tasks (like Hudi/Iceberg metadata or Java UDFs), this internal JVM is not configured to log GC to a separate file by default. Instead: * Default Behavior: Any output from the internal JVM (including GC info if enabled) is usually piped to

stderr

stdout

, which ends up in the cn.out
file. * In EKS/Kubernetes: Since you have redirected logs to stdout, any JVM-related logs from the CN should be visible in your container logs (

kubectl logs <cn-pod>

) alongside the standard

<http://cn.INFO|cn.INFO>

output. 3. How to verify or enable CN GC logging If you specifically need to see GC logs for the CN's internal JVM or want to force them to a file: 1. Check the

cn.conf

(usually via a ConfigMap in EKS). 2. Look for the

env JAVA_OPTS

JAVA_OPTS

variable. 3. If you want to enable file-based GC logging for the CN, you would add the JVM logging flags there:

Copy code

bash
    JAVA_OPTS="-Xlog:gc*:/opt/starrocks/cn/log/cn.gc.log:time"

Note: If this is not present, the CN internal JVM will simply follow the default output stream. Summary * FE logs on disk: These are JVM-native logs that bypass your Log4j stdout settings. * CN GC logs: These don't exist as a separate file by default; they are typically merged into the standard output (

cn.out

) or the container's stdout in EKS. References • administration/management/logs.md • administration/management/FE_configuration.md • administration/management/BE_configuration.md • deployment/helm.md References • administration/management/logs.md

Rumit Jain

07/15/2026, 7:13 AM

@Rocky After the v4.1.1 upgrade i can see data in starrocks_fe_db_size_bytes metrics but data in this metric was not there in v 4.0.1. Is this something newly added in v 4.1.1?

Rocky

07/15/2026, 7:13 AM

Yes, the starrocks_fe_db_size_bytes
metric is indeed a new addition (or newly exposed) in StarRocks v4.1. In previous versions like v4.0.1, database size information was generally only available by manually running SQL commands such as

SHOW DATA

. While this provided the size on disk, it was not exported as a standard Prometheus gauge, making it difficult to monitor database growth over time or set alerts in Grafana. Key Changes in v4.1.1: * Enhanced Observability: As part of the v4.1 release, StarRocks introduced several new metrics to improve visibility into storage and metadata, particularly for shared-data (cloud-native) clusters. * Database-Level Tracking: This specific metric allows the FE (Front End) to report the total size of data for each database directly to Prometheus. It typically includes a

db_name

label so you can track storage consumption per database. * Cost Efficiency: In the shared-data architecture (where data resides on S3/OSS), tracking storage size at the metadata layer (FE) is more efficient than the legacy methods used in shared-nothing architectures. Why you didn't see it in v4.0.1: In the 4.0.x series, the FE metrics primarily focused on query performance, connections, and JVM health. Storage metrics were often decentralized across BE nodes or required explicit metadata crawls that were not triggered by the Prometheus scrape interval. Related Metrics in v4.1: Along with

starrocks_fe_db_size_bytes

, you may also see other new storage-related metrics in your v4.1.1 cluster: * starrocks_fe_tablet_num
: Now more accurately tracks tablets in shared-data environments. * starrocks_fe_catalog_recycle_bin_size_bytes
: Tracks the size of data currently held in the catalog recycle bin (introduced in recent 4.1.x patches). If you are using the official StarRocks Grafana dashboard, you may need to update it to the latest version to see these new metrics visualized automatically. References • administration/management/monitoring/metrics.md • release_notes/release-4.1.md • administration/management/monitoring/Monitor_and_Alert.md • sql-reference/information_schema/fe_metrics.md References • administration/management/monitoring/metrics.md • release_notes/release-4.1.md • sql-reference/information_schema/fe_metrics.md References • administration/management/monitoring/metrics.md • sql-reference/information_schema/fe_metrics.md

Rumit Jain

07/15/2026, 7:19 AM

@Rocky Whatis the unit in which starrocks_fe_db_size_bytes exposes the data?

Rocky

07/15/2026, 7:20 AM

The metric

starrocks_fe_db_size_bytes

exposes data in Bytes. Following the standard Prometheus naming convention used across StarRocks, any metric with the suffix

_bytes

provides a raw count in bytes. Key Details: * Unit: Bytes (B). To convert this to GiB in tools like Grafana, you would divide the value by $1024^3$ (or $1,073,741,824$). * Logic: This metric typically reflects the total storage size of all tablets belonging to a specific database as tracked by the FE's metadata. * Update Frequency: The value is updated periodically based on the metadata reporting interval (governed by the FE configuration

db_used_data_quota_update_interval_secs

, which defaults to 300 seconds). If you are using this in a Grafana dashboard, you should set the Unit for the Y-axis to

Data / bytes (IEC)

to automatically see the values formatted as KiB, MiB, or GiB. References • administration/management/monitoring/metrics.md • administration/management/FE_configuration.md

Rumit Jain

07/15/2026, 7:48 AM

@Rocky What is enable_push_down_schema in v4.1.1?

Rocky

07/15/2026, 7:48 AM

In StarRocks v4.1.1, enable_push_down_schema
is a new INSERT property used to optimize data ingestion when using the

FILES()

table function. It is designed to simplify and harden the process of loading data from cloud storage (like S3 or HDFS) by aligning the source file schema directly with the target table's metadata. Key Functions * Schema Alignment: When set to

true

, StarRocks "reshapes" the schema of the

FILES()

function to match the target table. It automatically trims extra columns from the files and adds missing columns (filling them with default values) to match the target table's structure. * *Improved

SELECT *

Behavior:* In

BY NAME

mode, if you use

SELECT _ FROM FILES(...)

, the

expands to the target table's column names rather than the names inferred from the source files. * Type Safety: It pushes down the target table's column types to the file scanner. This reduces "looseness" or errors caused by automatic type inference from files (e.g., preventing a column inferred as a string from failing when the target table requires a specific numeric type). Usage Example You apply this property within the

PROPERTIES

clause of an

INSERT

statement:

Copy code

sql
INSERT INTO target_table
PROPERTIES ("enable_push_down_schema" = "true")
SELECT * FROM FILES(
    "path" = "<s3://bucket/data.parquet>",
    "format" = "parquet"
);

Important Notes * Exclusivity: This property is mutually exclusive with the explicit

schema

parameter inside the

FILES()

function. You cannot use both at the same time. * Version Context: While related to the earlier

files_enable_insert_push_down_schema

FE configuration,

enable_push_down_schema

as an INSERT property provides more granular, per-query control over this behavior. * Supported Formats: It is primarily used with structured and semi-structured formats like Parquet and ORC where schema mapping is critical. References • release_notes/release-4.1.md • sql-reference/sql-statements/table_bucket_part_index/FILES.md • administration/management/FE_configuration.md (for related

files_enable_insert_push_down_schema

info`) References • administration/management/FE_configuration.md • sql-reference/System_variable.md • administration/management/BE_configuration.md

Rumit Jain

07/16/2026, 12:44 PM

@Rocky What are these error in one CN pods out of 2 running in EKS?

Copy code

I20260716 12:43:53.786630 281468154542592 heartbeat_server.cpp:107] Updating master info: TMasterInfo(network_address=TNetworkAddress(hostname=kube-starrocks-fe-0.kube-starrocks-fe-search.prod.svc.cluster.local, port=9020), cluster_id=1533887730, epoch=1, token=<hidden>, backend_ip=kube-starrocks-cn-1.kube-starrocks-cn-search.prod.svc.cluster.local, http_port=8030, heartbeat_flags=0, backend_id=10085, min_active_txn_id=103091, run_mode=SHARED_DATA, disabled_disks=<null>, decommissioned_disks=<null>, encrypted=0, stop_regular_tablet_report=1, node_type=Compute)
W20260716 12:43:56.639477 281468653337088 socket.cpp:1379] Fail to wait EPOLLOUT of fd=332: Connection timed out [110]
I20260716 12:43:58.137608 281472604764672 daemon.cpp:140] Current memory statistics: process(458395536), query_pool(0), load(0), metadata(80306846), compaction(0), schema_change(0), page_cache(2450216), update(0), passthrough(0), clone(0), consistency(0), datacache(2450216), jit(0)
W20260716 12:44:00.139735 281468681845248 socket.cpp:1379] Fail to wait EPOLLOUT of fd=332: Connection timed out [110]

Rocky

07/16/2026, 12:45 PM

These logs indicate a network connectivity issue between that specific CN pod and the Leader FE node. The error

Fail to wait EPOLLOUT: Connection timed out [110]

means the CN pod attempted to send data to another node (likely the FE) over a network socket, but the operation timed out at the TCP level because the destination did not acknowledge the data in time. Why this is happening in only one pod: Since one pod is healthy and the other is not, the issue is likely not a global cluster configuration but rather a local environment issue on that specific EKS node or pod. 1. Inter-Pod Networking (EKS/Kubernetes) The CN pod is trying to communicate with

kube-starrocks-fe-0...:9020

. * Check CoreDNS: Occasionally, a specific pod can have trouble resolving internal service DNS. Although the log shows the full hostname, check if

nslookup

from inside the failing pod is slow. * Security Groups/Network Policies: Ensure there isn't a restrictive NetworkPolicy or AWS Security Group preventing traffic between the specific EKS worker node where this CN is running and the FE node. * MTU Mismatch: If your EKS cluster uses a CNI like Calico or has specific VPC settings, an MTU mismatch can cause larger packets (like those containing heartbeat or metadata updates) to be dropped, while smaller "ping" packets pass. 2. Resource Exhaustion or Throttling * CPU Throttling: If the CN pod is hitting its CPU limit, the networking stack (BRPC) may not be able to process the

EPOLL

events fast enough, leading to timeouts. Check the

container_cpu_usage_seconds_total

metric. * Node Port Exhaustion: If the worker node is running many pods, it might be out of available ephemeral ports or hitting

conntrack

limits. 3. Zombie Connections The

fd=332

indicates a specific file descriptor (socket). If the FE was restarted or the network was interrupted, the CN might be holding onto a "half-open" connection. * Action: Restart the failing CN pod. If it’s a temporary networking glitch, the restart will force it to establish fresh connections. Troubleshooting Steps 1. Check FE Logs: Search the Leader FE logs for this CN's IP address. Look for messages like

backend [10085] is dead

heartbeat failed

. 2. Test Connectivity: Execute into the failing CN pod and try to reach the FE port:

Copy code

bash
    kubectl exec -it <cn-pod-name> -- curl -v <http://kube-starrocks-fe-0.kube-starrocks-fe-search.prod.svc.cluster.local:8030/api/bootstrap>

If this times out or is slow, you have a confirmed networking path issue. 3. Check Node Health: See if the failing pod is on a different AWS EC2 instance than the working one. If so, inspect the health and network metrics of that specific EC2 instance in the AWS Console. Is it a bug? There is a known issue (tracked in GitHub #51912) where these logs can appear excessively in K8s environments if the network is jittery. While often harmless if the node stays "Alive" in

SHOW BACKENDS

, it indicates the underlying network is not as stable as StarRocks expects. References • administration/management/monitoring/Monitor_and_Alert.md • faq/operation_maintenance_faq.md • deployment/helm.md (for EKS network requirements) References • administration/management/monitoring/Monitor_and_Alert.md • faq/operation_maintenance_faq.md • administration/management/BE_configuration.md References • faq/operation_maintenance_faq.md

Rocky

07/16/2026, 12:45 PM

• administration/management/FE_configuration.md • knowledge/trouble_shooting/fe_deadlock_problems.md

Rumit Jain

07/16/2026, 12:52 PM

@Rocky Show compute nodes still shoing both the CN pods as ALIVE

Copy code

show compute nodes\G;
*************************** 1. row ***************************
        ComputeNodeId: 10053
                   IP: kube-starrocks-cn-0.kube-starrocks-cn-search.prod.svc.cluster.local
        HeartbeatPort: 9050
               BePort: 9060
             HttpPort: 8040
             BrpcPort: 8060
        LastStartTime: 2026-07-14 10:37:21
        LastHeartbeat: 2026-07-16 12:42:58
                Alive: true
 SystemDecommissioned: false
ClusterDecommissioned: false
               ErrMsg:
              Version: 4.0.1-cd9df36
             CpuCores: 8
             MemLimit: 12.960GB
    NumRunningQueries: 0
           MemUsedPct: 4.22 %
           CpuUsedPct: 0.0 %
     DataCacheMetrics: Status: Normal, DiskUsage: 2.9GB/799.6GB, MemUsage: 3.3MB/2.5GB
       HasStoragePath: true
           StatusCode: OK
          StarletPort: 9070
             WorkerId: 1
        WarehouseName: default_warehouse
            TabletNum: 151
*************************** 2. row ***************************
        ComputeNodeId: 10085
                   IP: kube-starrocks-cn-1.kube-starrocks-cn-search.prod.svc.cluster.local
        HeartbeatPort: 9050
               BePort: 9060
             HttpPort: 8040
             BrpcPort: 8060
        LastStartTime: 2026-07-14 10:34:21
        LastHeartbeat: 2026-07-16 12:42:58
                Alive: true
 SystemDecommissioned: false
ClusterDecommissioned: false
               ErrMsg:
              Version: 4.0.1-cd9df36
             CpuCores: 8
             MemLimit: 12.960GB
    NumRunningQueries: 0
           MemUsedPct: 3.29 %
           CpuUsedPct: 0.0 %
     DataCacheMetrics: Status: Normal, DiskUsage: 2.2GB/799.6GB, MemUsage: 2.3MB/2.5GB
       HasStoragePath: true
           StatusCode: OK
          StarletPort: 9070
             WorkerId: 5
        WarehouseName: default_warehouse
            TabletNum: 151
2 rows in set (0.01 sec)

ERROR:
No query specified

Rocky

07/16/2026, 12:53 PM

The errors you are seeing in your CN pod logs (

Fail to wait EPOLLOUT: Connection timed out [110]

) indicate a network communication failure specifically on the bRPC channel (typically port

), even though the node still appears

ALIVE

in the FE's metadata. Why the node is still "ALIVE" StarRocks uses different mechanisms for "liveness" and "data transfer": * Liveness (Heartbeat): The FE sends a heartbeat to the CN (port

). If this succeeds, the node is marked as

ALIVE

. * Data Transfer (bRPC): The error in your logs comes from

socket.cpp

, which belongs to the bRPC framework used for transferring query data and metadata updates. Your CN is successfully receiving heartbeats (as seen in the

Updating master info

log), but it is timing out when trying to send data back to the FE or other nodes. This creates a "gray failure" where the node is technically in the cluster but may cause query timeouts or "Deadline Exceeded" errors. Likely Causes in EKS Since only one of your two CN pods is affected, this is almost certainly a pod-level or node-level network issue rather than a StarRocks software bug: 1. TCP Connection Stalling: The specific file descriptor (

fd=332

) has a stuck outbound buffer. This often happens if the destination (the FE) is not reading from the socket fast enough or if the underlying TCP connection was silently dropped by an AWS Network Load Balancer (NLB) or security group without a proper

FIN

packet. 2. EKS Node Resource Limits: If the EKS worker node hosting

kube-starrocks-cn-1

is under heavy load, the Linux kernel's

epoll

mechanism may delay processing, leading to the

EPOLLOUT

timeout. 3. MTU Mismatch: In some AWS VPC configurations (especially with Jumbo Frames), if the CN tries to send a large packet that exceeds the path MTU, it will be dropped, causing the socket to wait indefinitely for an ACK and eventually time out. Recommended Actions 1. Quick Fix: Restart the Pod The simplest way to clear a stuck file descriptor is to delete the affected CN pod. Kubernetes will recreate it, forcing a fresh set of network connections.

Copy code

bash
kubectl delete pod kube-starrocks-cn-1 -n <namespace>

2. Verify Network Path From within the failing CN pod, test if you can reach the FE's bRPC port:

Copy code

bash
kubectl exec -it kube-starrocks-cn-1 -- nc -zv kube-starrocks-fe-0.kube-starrocks-fe-search.prod.svc.cluster.local 8060

3. Adjust bRPC Buffer (If under heavy load) If you see these errors frequently during high-concurrency queries, you can increase the unwritten outbound buffer in your

cn.conf

. The default is 1GB. * Parameter:

brpc_socket_max_unwritten_bytes

* Suggested Value:

4294967296

(4GB) _ _Note: This requires a restart of the CN service to take effect.* 4. Known Logging Issue There is a documented issue (GitHub #51912) where these logs appear excessively in Kubernetes environments due to how bRPC handles idle connections. If your queries are completing successfully and the

LastHeartbeat

SHOW COMPUTE NODES

remains current, these logs may be noisy but harmless. However, if you see query failures, prioritize the networking checks above. References • administration/management/BE_configuration.md • faq/operation_maintenance_faq.md • sql-reference/sql-statements/cluster-management/nodes_processes/SHOW_COMPUTE_NODES.md • GitHub Issue #51912: Excessive EPOLLOUT logs in CN Pods References • knowledge/usage/best-practice.md • administration/management/BE_configuration.md

Rocky

07/16/2026, 12:53 PM

• faq/Sql_faq.md • faq/Deploy_faq.md • administration/management/FE_configuration.md • faq/operation_maintenance_faq.md

Rumit Jain

07/17/2026, 11:39 AM

@Rocky In v4.1.1 along with fe proxy in fe+CN, I am triying the load job manually but timestamp is coming as Singaporetimezone even though on starrocks the variables are as below: | system_time_zone | UTC | | time_zone | UTC |

Rocky

07/17/2026, 11:40 AM

Even though you have set your session variables

time_zone

and

system_time_zone

UTC

, StarRocks loading processes often rely on specific job properties or global settings that default to

Asia/Shanghai

(which is the same offset as Singapore Time, UTC+8). In v4.1.1, here are the most likely reasons and fixes for why your manual load is still resulting in Singapore/Shanghai timestamps: 1. Loading Method Ignores Session Variables Most StarRocks loading methods (except

INSERT INTO ... SELECT

) do not use the session-level

time_zone

variable. They have their own internal default of

Asia/Shanghai

. * Broker Load: You must explicitly set the

timezone

in the

PROPERTIES

clause of the load statement.

Copy code

sql
    LOAD LABEL my_label
    (DATA INFILE("s3://...") INTO TABLE my_table)
    WITH BROKER s3_broker
    PROPERTIES ("timezone" = "UTC"); -- This is required

* Stream Load: You must pass the

timezone

in the HTTP header.

Copy code

bash
    curl --location-trusted -u user:passwd \
        -H "timezone: UTC" \
        -T data.csv <http://fe_host:http_port/api/db/table/_stream_load>

* Routine Load: Set

timezone

in the

job_properties

. 2. FE Proxy and Session Persistence If you are using the FE Proxy feature (introduced to route load traffic), and you are performing an

INSERT INTO ... SELECT FROM FILES()

, the session variable

SET time_zone = 'UTC'

might not be persisting across the connection handled by the proxy or being passed correctly to the Compute Nodes (CN). * Fix: Set the timezone globally to ensure all background load tasks and proxied sessions use UTC:

Copy code

sql
    SET GLOBAL time_zone = 'UTC';

Note: You may need to reconnect for this to reflect in your current session. 3. Parquet Behavior Change in v4.1.1 In StarRocks v4.0.12 and v4.1.1+, there is a significant change in how Parquet timestamps are handled: * Timezone-Naive Timestamps: If your Parquet file was written with

isAdjustedToUTC=false

(standard for many Spark/Hive exports), StarRocks now reads these as "wall-clock" values. It no longer shifts them by the session timezone. * The Issue: If your source data was originally in SGT and written to Parquet as "naive," StarRocks will load the numbers exactly as they appear in the file. If you then query them in a UTC session, they will appear to be shifted by 8 hours relative to what you expect. 4. Check the "Push Down Schema" Property Since you mentioned v4.1.1, if you are using the new

enable_push_down_schema

property with

FILES()

, it pushes the target table's metadata down to the scanner. If the target table column is

DATETIME

(which is timezone-agnostic in StarRocks), the scanner will convert the source file's

TIMESTAMP

DATETIME

using the FE's default timezone if the session context is missing. Summary Checklist 1. Run
SET GLOBAL time_zone = 'UTC';
to eliminate session-loss issues. 2. If using Broker Load, add

"timezone" = "UTC"

to your

PROPERTIES

. 3. Check if your source files are Parquet. If they are, check if they are "UTC adjusted." If they are not adjusted, StarRocks will read the "wall-clock" time without conversion. 4. Verify that your FE and CN pod system times are actually synchronized. While StarRocks reports

system_time_zone

, it's best to verify with

date

inside the pod. References • administration/management/timezone.md • sql-reference/sql-statements/loading_unloading/BROKER_LOAD.md • release_notes/release-4.1.md (Section on Parquet behavior changes) • loading/loading_introduction/feature-support-loading-and-unloading.md References

Rocky

07/17/2026, 11:40 AM

• administration/management/timezone.md • faq/loading/Broker_load_faq.md • loading/loading_introduction/feature-support-loading-and-unloading.md • sql-reference/sql-statements/loading_unloading/BROKER_LOAD.md • sql-reference/System_variable.md • sql-reference/sql-statements/loading_unloading/routine_load/CREATE_ROUTINE_LOAD.md • integrations/loading_tools/DataX-starrocks-writer.md • loading/Spark-connector-starrocks.md • faq/Sql_faq.md • deployment/environment_configurations.md • sql-reference/sql-functions/table-functions/files.md • sql-reference/sql-statements/loading_unloading/INSERT.md • sql-reference/data-types/semi_structured/VARIANT.md • data_source/feature-support-data-lake-analytics.md • unloading/Spark_connector.md • administration/management/FE_configuration.md • administration/management/BE_configuration.md • sql-reference/sql-statements/cluster-management/config_vars/SET.md • sql-reference/sql-statements/table_bucket_part_index/FILES.md • administration/management/monitoring/Monitor_and_Alert.md • faq/operation_maintenance_faq.md • deployment/helm.md • knowledge/trouble_shooting/fe_deadlock_problems.md • knowledge/usage/best-practice.md • faq/Deploy_faq.md • sql-reference/sql-statements/cluster-management/nodes_processes/SHOW_COMPUTE_NODES.md • release_notes/release-4.1.md • sql-reference/sql-statements/table_bucket_part_index/FILES.md • administration/management/FE_configuration.md • administration/management/FE_configuration.md • sql-reference/System_variable.md • administration/management/BE_configuration.md

Rocky

07/17/2026, 11:40 AM

• administration/management/monitoring/Monitor_and_Alert.md • faq/operation_maintenance_faq.md • administration/management/BE_configuration.md • faq/operation_maintenance_faq.md • administration/management/FE_configuration.md • knowledge/trouble_shooting/fe_deadlock_problems.md • administration/management/BE_configuration.md • faq/operation_maintenance_faq.md • sql-reference/sql-statements/cluster-management/nodes_processes/SHOW_COMPUTE_NODES.md • knowledge/usage/best-practice.md • administration/management/BE_configuration.md • faq/Sql_faq.md • faq/Deploy_faq.md • administration/management/FE_configuration.md • faq/operation_maintenance_faq.md References • administration/management/timezone.md • faq/loading/Broker_load_faq.md • sql-reference/sql-statements/loading_unloading/BROKER_LOAD.md • loading/Spark-connector-starrocks.md • integrations/loading_tools/DataX-starrocks-writer.md • sql-reference/sql-statements/loading_unloading/routine_load/CREATE_ROUTINE_LOAD.md • loading/loading_introduction/feature-support-loading-and-unloading.md • faq/Sql_faq.md

Rumit Jain

07/17/2026, 12:27 PM

@Rocky While inserting records using fe_proxy stream load api, why I am getting this?

Copy code

curl --location-trusted -u sr_admin_user             \
    -T ./NYPD_Crash_Data.csv                \
    -H "label:crashdata-0"                  \
    -H "column_separator:,"                 \
    -H "skip_header:1"                      \
    -H>  "enclose:\""                         \
    -H "max_filter_ratio:1"                 \
    -H "columns:tmp_CRASH_DATE, tmp_CRASH_T> IME, CRASH_DATE=str_to_date(concat_ws(' ', tmp_CRASH_DATE, tmp_CRASH_TIME), '%m/> %d/%Y %H:%i'),BOROUGH,ZIP_CODE,LATITUDE,LONGITUDE,LOCATION,> ON_STREET_NAME,CROSS_STREET_NAME,OFF_STREET_NAME,NUMBER_OF_PERSONS_INJURED,> NUMBER_OF_PERSONS_KILLED,NUMBER_OF_PEDESTRIANS_INJURED,NUMBER_OF_PEDESTRIAN> S_KILLED,NUMBER_OF_CYCLIST_INJURED,NUMBER_OF_CYCLIST_KILLED,NUMBER_OF_MO> TORIST_INJURED,NUMBER_OF_MOTORIST_KILLED,CONTRIBUTING_FACTOR_VEHICLE_1,CONTRIBUTING_FACTOR_VEHICLE_2,CONTRIBUTING_FACTOR_VEHICLE_3,CONTRIBUTING_FACTOR_VEHICLE_4,CONTRIBUTING_FACTOR_VEHICLE_5,COLLISION_ID,VEHICLE_TYPE_CODE_1,VEHICLE_TYPE_CODE_2,VEHICLE_TYPE_CODE_3,VEHICLE_TYPE_CODE_4,VEHICLE_TYPE_CODE_5" \
    -XPUT <http://kube-starrocks-fe-proxy-service:8080/api/quickstart/crashdata/_stream_load>>
Enter host password for user 'sr_admin_user':
<html>
<head><title>502 Bad Gateway</title></head>
<body>
<center><h1>502 Bad Gateway</h1></center>
<hr><center>nginx/1.28.0</center>
</body>
</html>

In fe-proxy pods these logs are coming:

Copy code

2026/07/17 12:21:46 [error] 23#23: *147974 recv() failed (104: Connection reset by peer) while reading response header from upstream, client: 127.0.0.6, server: , request: "PUT /api/quickstart/crashdata/_stream_load HTTP/1.1", upstream: "<http://10.218.106.23:8040/api/quickstart/crashdata/_stream_load>", host: "kube-starrocks-fe-proxy-service:8080"

Rocky

07/17/2026, 12:31 PM

The

502 Bad Gateway

and the

Connection reset by peer

error in your

fe-proxy

logs indicate that the CN pod (at IP 10.218.106.23) is forcibly closing the connection while the proxy is trying to forward your Stream Load data. This is almost certainly connected to the

EPOLLOUT

network timeouts you reported earlier in that same CN pod. Here is the breakdown of why this is failing and how to fix it: 1. The "Gray Failure" Connection Your previous logs showed one CN pod (

kube-starrocks-cn-1

) having socket timeouts. * The

fe-proxy

is attempting to send the CSV data to that CN's HTTP port (

). * Because the CN's networking stack is already struggling (the

EPOLLOUT

errors), it is likely failing to buffer the incoming stream or its internal web server (Mongoose/bRPC) is crashing/restarting the thread, leading to the

Connection reset by peer

. Action: Check if

10.218.106.23

is the IP of the CN pod that was showing errors. If it is, restart that CN pod immediately.

Copy code

bash
kubectl delete pod <name-of-cn-pod-at-10.218.106.23>

2. Header and Timezone Issues In your

curl

command, you have a few syntax issues that might be causing the backend to reject the request: * Timezone: As discussed previously, StarRocks defaults to

Asia/Shanghai

. If your

CRASH_DATE

string doesn't match the expected format or timezone logic, it could trigger errors during the transformation. * Recommendation: Add the timezone header explicitly to your

curl

to ensure it matches your session expectation:

Copy code

bash
    -H "timezone: UTC"

3. Nginx Proxy Configuration (
client_max_body_size
) The

502 Bad Gateway

from

nginx/1.28.0

can also occur if the file is large and the proxy's buffer is exceeded. * If your

NYPD_Crash_Data.csv

is large, ensure the

fe-proxy

(Nginx) is configured with:

Copy code

nginx
    client_max_body_size 0;
    proxy_request_buffering off;

* Without

proxy_request_buffering off

, Nginx tries to read the entire file into its own disk/memory before sending it to StarRocks, which often causes timeouts or resets in Kubernetes environments. 4. Directing to FE instead of Proxy As a troubleshooting step, try to bypass the

fe-proxy

and hit the FE service directly. This will confirm if the issue is with the proxy or the CNs:

Copy code

bash
# Change the URL to the FE service port 8030
curl --location-trusted -u sr_admin_user ... \
    -XPUT <http://kube-starrocks-fe-service:8030/api/quickstart/crashdata/_stream_load>

Note: The FE will send a
307 Temporary Redirect
to a CN. If the CN is healthy, the load will proceed. Summary Checklist 1. Restart the failing CN pod. The

104: Connection reset

is a symptom of the same network issue causing the

EPOLLOUT

logs. 2. Verify Upstream IP: Run

kubectl get pods -o wide

to confirm if

10.218.106.23

is the "bad" CN pod. 3. Fix Curl Syntax: Ensure your

-H

flags don't have trailing

characters (which look like copy-paste artifacts in your snippet). 4. Add Timezone: Include

-H "timezone: UTC"

to ensure your

str_to_date

calculation behaves as expected. References • loading/StreamLoad.md • faq/loading/Stream_load_faq.md • administration/management/BE_configuration.md (Check

brpc_socket_max_unwritten_bytes

) References • loading/StreamLoad.md • faq/loading/Stream_load_faq.md • administration/management/BE_configuration.md • sql-reference/sql-statements/loading_unloading/STREAM_LOAD.md

Rocky

07/17/2026, 12:31 PM

Rocky

07/17/2026, 12:31 PM

Rocky

07/17/2026, 12:31 PM

Rocky

07/17/2026, 12:31 PM

Rocky

07/17/2026, 12:31 PM

Rocky

07/17/2026, 12:31 PM

Rocky

07/17/2026, 12:31 PM

• faq/Deploy_faq.md • administration/management/FE_configuration.md • faq/operation_maintenance_faq.md • loading/StreamLoad.md • faq/loading/Stream_load_faq.md • administration/management/BE_configuration.md • sql-reference/sql-statements/loading_unloading/STREAM_LOAD.md • faq/operation_maintenance_faq.md • deployment/helm.md • administration/management/monitoring/Monitor_and_Alert.md • knowledge/usage/best-practice.md • faq/Deploy_faq.md • administration/management/FE_configuration.md • sql-reference/sql-statements/cluster-management/nodes_processes/SHOW_COMPUTE_NODES.md • release_notes/release-4.1.md • administration/management/timezone.md • sql-reference/sql-statements/table_bucket_part_index/FILES.md • sql-reference/System_variable.md • faq/Sql_faq.md • loading/loading_introduction/feature-support-loading-and-unloading.md • sql-reference/sql-statements/loading_unloading/BROKER_LOAD.md • faq/loading/Broker_load_faq.md • knowledge/trouble_shooting/fe_deadlock_problems.md • sql-reference/sql-statements/cluster-management/config_vars/SET.md • sql-reference/sql-functions/table-functions/files.md • sql-reference/sql-statements/loading_unloading/INSERT.md • sql-reference/data-types/semi_structured/VARIANT.md • data_source/feature-support-data-lake-analytics.md • unloading/Spark_connector.md • sql-reference/sql-statements/loading_unloading/routine_load/CREATE_ROUTINE_LOAD.md • integrations/loading_tools/DataX-starrocks-writer.md • loading/Spark-connector-starrocks.md • deployment/environment_configurations.md

Rocky

07/17/2026, 12:31 PM

Rocky

07/17/2026, 12:31 PM

Rumit Jain

07/20/2026, 5:38 AM

@Rocky I tried triggering the load job manually and with the below query along with timezone property but still the job got created with singaporeTimeZone somehow!:

Copy code

LOAD LABEL sampledatameshdb.sampledatameshdb__testdatameshlakehouseobject__messagebasedbundle__f84f4448__c7kg6__1__1784276718221_BMTestUTCTZ (
 DATA INFILE("<s3://ABC-PWR-dnp-QQQ/QQQ/default/KLM/ownerid=00000000-0000-0000-0000-000000000000/failures/XYZ__f84f4448__c7kg6__1__1784276718221_BMTestUTCTZ.parquet>")
 INTO TABLE TABLE_NAME
)
WITH BROKER (
 "aws.s3.use_instance_profile" = "true",
 "aws.s3.use_aws_sdk_default_behavior" = "true",
 "aws.s3.region" = "us-east-1"
)
PROPERTIES (
 "timeout" = "3600",
 "timezone" = "UTC"
);

I am running v4.1.1 FE+CN in AWS EKS.

Rocky

07/20/2026, 5:43 AM

In StarRocks v4.1.1, there are two distinct reasons why you might see a "Singapore Time" (UTC+8) offset despite setting the

"timezone" = "UTC"

property in your Broker Load. 1. The Parquet "Wall-Clock" Behavior Change (Most Likely) Starting in v4.0.12 and v4.1.1, StarRocks changed how it reads timestamps from Parquet files to align with Spark and Trino. * How it works now: StarRocks checks the

isAdjustedToUTC

metadata attribute in the Parquet file. * If
isAdjustedToUTC = false
(Timezone-naive): StarRocks now reads these as "wall-clock" values. It ignores your job's

timezone

property and does not perform any shift. * The Issue: If your source data was generated by a system in Singapore Time (or a machine with a UTC+8 default) and written as a naive timestamp, StarRocks will load the "10:00 AM" in the file as "10:00 AM" in the table. When you query it in a UTC session, it will look like it has an 8-hour shift compared to your expectations. * Fix: Check how your Parquet files are being written. If they are timezone-naive, you may need to use a transformation in your

LOAD

statement to adjust them manually (e.g.,

SET (my_col = seconds_sub(my_col, 28800))

). 2.
SHOW LOAD
Metadata Display If the "Singapore Time" you are seeing is the

CreateTime

FinishTime

in the output of

SHOW LOAD

, this is a display-only behavior of the Frontend (FE). * The metadata for load jobs (timestamps of when the job started/finished) is displayed using the FE's
system_time_zone
. * In many StarRocks Docker/EKS environments, the default is

Asia/Shanghai

(UTC+8), which is identical to Singapore Time. * Fix: To force the FE to display these metadata timestamps in UTC, you must ensure the global variable is set and the FE pod itself is running in UTC.

Copy code

sql
    SET GLOBAL time_zone = 'UTC';

Note: After running this, reconnect your SQL client to see the change. 3. Verification Steps To pinpoint where the "Singapore" setting is coming from, run the following: 1. Check FE Pod Time: Exec into your FE pod and run

date

. If it shows

+0800

, the FE's internal clock is the source of the metadata display shift. 2. Inspect Parquet Metadata: Use a tool like

parquet-meta

to see if your timestamp columns have

isAdjustedToUTC: true

. If it is

false

, the

"timezone" = "UTC"

property in your

LOAD

statement is effectively being ignored for data conversion by design in v4.1.1. 3. Use Explicit Offset: Try using the numeric offset in your

PROPERTIES

to rule out name-resolution issues:

Copy code

sql
    PROPERTIES (
     "timezone" = "+00:00"
    );

Summary Recommendation If your data values are wrong: Use a transformation or check the

isAdjustedToUTC

flag in your source Parquet files. If the job status time is wrong: Run

SET GLOBAL time_zone = 'UTC';

and verify the

TZ

environment variable in your Helm chart for the FE/CN pods. References • release_notes/release-4.1.md (Section on Parquet Behavior Changes) • administration/management/timezone.md • sql-reference/sql-statements/loading_unloading/BROKER_LOAD.md • sql-reference/sql-functions/table-functions/files.md (Details on Parquet timestamp mapping) References • administration/management/timezone.md • release_notes/release-4.1.md • sql-reference/sql-statements/loading_unloading/BROKER_LOAD.md

Rocky

07/20/2026, 5:43 AM

Rocky

07/20/2026, 5:43 AM

Rocky

07/20/2026, 5:43 AM

Rocky

07/20/2026, 5:43 AM

Rocky

07/20/2026, 5:43 AM

Rocky

07/20/2026, 5:43 AM

Rocky

07/20/2026, 5:43 AM

Rocky

07/20/2026, 5:43 AM

Rocky

07/20/2026, 5:43 AM

Open in Slack

Previous Next