http://starrocks.io logo
Join Slack
Powered by
<@U0A71G31CDV> My tool to provision users, roles a...
# questions-and-troubleshooting
s
@Rocky My tool to provision users, roles and grants did the initial 
root
password change itself after connecting using the default empty password. I did not use the root password init container feature to simply gitops manipulation. There has been a BE node rotation and the replacement BE pod now fails too boot because it doesn't like that change. The BE k8s pod log are looping like this which trigger crash loop. What is the proper way to change the root password so that the entrypoint scripts are not failing like this.
Copy code
[Wed Jun  3 12:01:26 UTC 2026] Process conf file be.conf ...
[Wed Jun  3 12:01:26 UTC 2026] Add myself (starrocks-insights-be-1.starrocks-insights-be-search.ucc-core.svc.cluster.local:9050) into FE ...
ERROR 1045 (28000): Access denied for user 'root' (using password: NO)
ERROR 1045 (28000): Access denied for user 'root' (using password: NO)
r
To resolve the boot loop where the BE pod fails with 
Access denied for user 'root' (using password: NO)
, you must provide the new root password to the BE's entrypoint script via the 
MYSQL_PWD
 environment variable. In StarRocks Kubernetes deployments (Operator or Helm), the BE entrypoint script uses the 
mysql
client to register the node with the FE (
ALTER SYSTEM ADD BACKEND ...
). When you change the password from the default empty value, the script fails unless it can authenticate. The Proper Way to Fix This You need to inject the 
MYSQL_PWD
environment variable into your StarRocks components using a Kubernetes Secret. 1. Create a Secret for the password If you haven't already, create a secret containing the new 
root
password:
Copy code
bash
kubectl create secret generic starrocks-root-pass --from-literal=password='YOUR_NEW_PASSWORD'
2. Update your Configuration Depending on how you deploy StarRocks, add the environment variable to your BE (and FE/CN) specs: If using the StarRocks Operator (StarRocksCluster CRD): Update your 
StarRocksCluster
YAML to include 
beEnvVars
. It is recommended to add this to 
feEnvVars
and 
cnEnvVars
as well to ensure all components can communicate.
Copy code
yaml
apiVersion: <http://starrocks.com/v1capsule|starrocks.com/v1capsule>
kind: StarRocksCluster
metadata:
  name: starrocks-insights
spec:
  starRocksBeSpec:
    beEnvVars:
      - name: "MYSQL_PWD"
        valueFrom:
          secretKeyRef:
            name: starrocks-root-pass
            key: password
  starRocksFeSpec:
    feEnvVars:
      - name: "MYSQL_PWD"
        valueFrom:
          secretKeyRef:
            name: starrocks-root-pass
            key: password
If using the Helm Chart: Add the environment variables to your `values.yaml`:
Copy code
yaml
starrocks:
  be:
    envVars:
      - name: MYSQL_PWD
        valueFrom:
          secretKeyRef:
            name: starrocks-root-pass
            key: password
  fe:
    envVars:
      - name: MYSQL_PWD
        valueFrom:
          secretKeyRef:
            name: starrocks-root-pass
            key: password
Why this happens The BE entrypoint script specifically looks for the 
MYSQL_PWD
environment variable. If it is present, the 
mysql
command used during the "Add myself into FE" phase will automatically use that password. Without it, the client attempts to connect without a password, triggering the 
1045 Access Denied
error you see in the logs. Once this environment variable is set, the replacement BE pod will be able to authenticate with the FE and complete its registration successfully. References • quick_start/helm.md • faq/operation_maintenance_faq.md • administration/user_privs/authentication/native_authentication.md • deployment/post_deployment_setup.md • deployment/deploy_manually.md
s
@Rocky What happen then if I create a brand new cluster and provide MYSQL_PWD variable already set, would that be the correct initial password?
r
No, providing the 
MYSQL_PWD
environment variable on a brand new cluster will not set the initial password, and it will likely cause the same boot loop error. Here is why: * 
MYSQL_PWD
is for the client: This variable tells the StarRocks helper scripts (the "client") what password to use when they try to talk to the FE. * The Database is separate: When a StarRocks cluster is first initialized, the actual 
root
user in the database always starts with an empty password by default. What happens if you set it on day one? If you provide 
MYSQL_PWD=YOUR_NEW_PASSWORD
to a brand new cluster: 1. The FE starts up with an empty 
root
password. 2. The BE starts up and tries to register itself with the FE using the password in 
MYSQL_PWD
. 3. The FE rejects the connection because it expects no password. 4. The BE enters the same 
Access denied
crash loop you are seeing now. The Correct GitOps Approach To initialize a cluster with a password automatically, you should use the 
initPassword
 feature in the Helm chart or Operator. This feature creates a one-time job (
kube-starrocks-initpwd
) that: 1. Connects to the new cluster using the default empty password. 2. Sets the 
root
password to the value found in your secret. 3. Updates the internal state so that subsequent BE/CN boots (using 
MYSQL_PWD
) will succeed. Helm Configuration:
Copy code
yaml
starrocks:
  initPassword:
    enabled: true
    passwordSecret: starrocks-root-pass # Must contain a 'password' key
Summary: * To set the password initially: Use the 
initPassword
feature. * To allow pods to boot after the password is set: Use the 
MYSQL_PWD
environment variable. (as described in the previous response). previous answer). References • quick_start/helm.md • deployment/post_deployment_setup.md • administration/user_privs/authentication/native_authentication.md • deployment/helm.md
s
@Rocky There is no other way force root password to bootstrap more cleanly without enabling/disabling init container and later injecting MYSQL_PWD ? This feels complicated operation wise.
r
I understand that the multi-step process feels cumbersome for GitOps. However, because of the architecture of StarRocks (FE and BE are separate processes), there isn't a single "bootstrap environment variable" like 
MYSQL_ROOT_PASSWORD
that the FE binary reads to set its own internal root password during first boot. To achieve a clean, single-step GitOps bootstrap, you should use the 
initPassword
 feature in the Helm chart or Operator. This is the intended "declarative" way to handle this. Why 
initPassword
is the "Clean" GitOps Way: When you enable this in your YAML, the StarRocks Operator manages the complexity for you: 1. It starts the FE and BE. 2. It automatically runs a one-time helper Job (
kube-starrocks-initpwd
). 3. This Job waits for the FE to be ready, connects with the default empty password, and sets it to the value in your Secret. 4. Crucially, you still need 
MYSQL_PWD
in your BE/FE environment variables so that when pods rotate, the entrypoint scripts can successfully run the 
ALTER SYSTEM
commands needed for node registration. The Most Streamlined YAML (One-and-Done) If you want to avoid "enabling/disabling" and just have a single GitOps manifest that works from Day 1 through Day 1000, use this pattern:
Copy code
yaml
starrocks:
  # 1. This handles the INITIAL change (once per cluster lifetime)
  initPassword:
    enabled: true
    passwordSecret: starrocks-root-pass

  # 2. These handle all SUBSEQUENT reboots and rotations
  starrocksFeSpec:
    feEnvVars:
      - name: MYSQL_PWD
        valueFrom:
          secretKeyRef:
            name: starrocks-root-pass
            key: password
  starrocksBeSpec:
    beEnvVars:
      - name: MYSQL_PWD
        valueFrom:
          secretKeyRef:
            name: starrocks-root-pass
            key: password
Is there any other way? If you absolutely cannot use the 
initPassword
feature, the only other "clean" way is to: 1. Bootstrap with authentication disabled: Set 
enable_auth_check = false
in your initial 
fe.conf
. 2. This allows BEs to join without any password. 3. Once the cluster is up, use your GitOps tool to run a one-time SQL script to set the password. 4. Update your config to 
enable_auth_check = true
and inject 
MYSQL_PWD
. Recommendation: The 
initPassword
approach is significantly less fragile than the manual auth-check toggle. By including both the 
initPassword
block and the 
MYSQL_PWD
environment variables in your initial Helm/Operator manifest, the cluster will bootstrap correctly and survive node rotations without any manual steps. References • quick_start/helm.md • administration/user_privs/authentication/native_authentication.md • deployment/helm.md References • quick_start/helm.md • administration/user_privs/authentication/native_authentication.md • deployment/post_deployment_setup.md • deployment/helm.md
s
@Rocky Can I safely let 
initPassword.enabled = true
after the initial boostrapping is done ?
r
Yes, you can safely leave 
initPassword.enabled = true
in your configuration after the initial bootstrap. In fact, the official StarRocks Helm documentation explicitly recommends that if you used 
initPassword
during the initial install, you should always retain that configuration in your 
values.yaml
for all future upgrades. Why it is safe: 1. Idempotency: The 
kube-starrocks-initpwd
job is designed to handle the "already initialized" state. If the password in the database already matches the password in your secret, the initialization script effectively becomes a no-op. 2. Lifecycle Management: In a Helm-based deployment, the 
initPassword
job is often managed as a Helm hook or a one-time resource. Leaving the flag enabled ensures that if you ever tear down the cluster and recreate it from the same GitOps state, the bootstrap logic is ready to run again. 3. Stability: Disabling it later in your GitOps pipeline adds unnecessary operational churn (an extra commit/PR) without providing any technical benefit, as the Job typically stays in a 
Completed
state or is ignored by the Operator once finished. Important Distinction for GitOps To maintain a stable environment, you must keep both configurations active: * 
initPassword
: Handles the first-time setup (Day 0). It connects with an empty password and sets the one in your secret. * 
MYSQL_PWD
(Env Var): Handles pod lifecycle (Day 1+). This is what your BE and CN pods use to authenticate with the FE every time they start up or rotate. Even if 
initPassword
is enabled, your BEs will still crash-loop on rotation if 
MYSQL_PWD
is missing. Summary of the "Set and Forget" Config This is the recommended setup for a permanent GitOps manifest:
Copy code
yaml
starrocks:
  # Leave this TRUE always. It ensures new clusters are bootstrapped.
  initPassword:
    enabled: true
    passwordSecret: starrocks-root-pass

  # Leave these ALWAYS. They ensure BE/CN pods can join the FE after restarts.
  starrocksBeSpec:
    beEnvVars:
      - name: MYSQL_PWD
        valueFrom:
          secretKeyRef:
            name: starrocks-root-pass
            key: password
Note on Password Rotation: If you need to change the password later, simply updating the secret and 
initPassword
value usually will not work via Helm upgrade (as the job only runs on initial deployment). You would need to change the password via SQL and then update the 
password
field in your Kubernetes Secret so that 
MYSQL_PWD
picks up the new value for future pod restarts. References • quick_start/helm.md • deployment/helm.md • administration/user_privs/authentication/native_authentication.md • deployment/post_deployment_setup.md References • administration/management/FE_configuration.md • quick_start/helm.md • deployment/post_deployment_setup.md • administration/user_privs/authentication/native_authentication.md • administration/management/BE_configuration.md • loading/Flink-connector-starrocks.md
s
@Rocky OK Got it. Now here's the catch, the K8S secret will get created indirectly by an 
<http://external-secrets.io/v1|external-secrets.io/v1>
resource I plan on adding in the 
starrocks.resources
freeform extra manifest. Will this resource get created initially ?
r
Using an 
ExternalSecret
(from 
<http://external-secrets.io|external-secrets.io>
) inside the 
starrocks.resources
field is a common GitOps pattern, but there is a specific timing "race" you need to be aware of. Will the resource be created initially? Yes. Anything you put in the 
starrocks.resources
field of the Helm chart is rendered as a standard Kubernetes manifest within the same Helm release. When you run 
helm install
, the 
ExternalSecret
object will be sent to the Kubernetes API at the same time as the StarRocks cluster resources. The "Secret Readiness" Catch The challenge is that an 
ExternalSecret
is just a "request" for a secret. The actual Kubernetes 
Secret
(which StarRocks needs) is created asynchronously by the External Secrets Operator after it fetches the value from your provider (AWS SM, HashiCorp Vault, etc.). Here is what will happen during the bootstrap: 1. Deployment: Helm creates the 
ExternalSecret
and the StarRocks 
initPassword
Job. 2. The Wait: The 
initPassword
Pod will try to start. Since it references the 
passwordSecret
(which doesn't exist yet because the External Secrets Operator is still working), the Pod will likely enter a 
CreateContainerConfigError
 or 
ContainerConfigError
 state. 3. Resolution: This is actually safe. Kubernetes will automatically retry starting the Pod. As soon as the External Secrets Operator finishes its sync and creates the 
Secret
object, the next K8s retry will succeed, the Pod will find the secret, and the 
initPassword
logic will run. Critical Configuration For this to work smoothly without manual intervention, ensure your 
ExternalSecret
is configured correctly: * Secret Name Match: The 
target.name
in your 
ExternalSecret
must exactly match the 
starrocks.initPassword.passwordSecret
name in your Helm values. * Key Name Match: The StarRocks script expects the password to be under a key named 
password
. Ensure your 
ExternalSecret
maps the external value to this specific key. Example 
ExternalSecret
in `starrocks.resources`:
Copy code
yaml
starrocks:
  initPassword:
    enabled: true
    passwordSecret: starrocks-root-pass # Must match 'target.name' below
  resources:
    - apiVersion: <http://external-secrets.io/v1beta1|external-secrets.io/v1beta1>
      kind: ExternalSecret
      metadata:
        name: starrocks-root-pass-es
      spec:
        refreshInterval: 1h
        secretStoreRef:
          name: vault-backend
          kind: ClusterSecretStore
        target:
          name: starrocks-root-pass # This is the K8s Secret that will be created
        data:
          - secretKey: password # StarRocks specifically looks for the 'password' key
            remoteRef:
              key: path/to/your/db/creds
              property: root_password
Summary * Is it safe? Yes. Kubernetes' native retry mechanism handles the delay while the External Secrets Operator syncs. * Operational Tip: If you see the 
kube-starrocks-initpwd
pod in an error state for the first 30 seconds, don't panic. It is just waiting for the external secret to materialize. Once it does, the pod will transition to 
Running
, then 
Completed
, and your cluster will be ready. References • quick_start/helm.md • deployment/helm.md • administration/user_privs/authentication/native_authentication.md • deployment/post_deployment_setup.md References • quick_start/helm.md • loading/Spark-connector-starrocks.md • administration/management/FE_configuration.md
s
@Rocky This is installed with argocd. I see some note about argocd and 
IsInstall
tell me more
r
Sorry, I encountered an error while processing your request: 400 INVALID_ARGUMENT. {'error': {'code': 400, 'message': 'The answer candidate length is too long with 8879 tokens, which exceeds the maximum token limit of 8192.', 'status': 'INVALID_ARGUMENT'}}
s
@Rocky Tell me about ArgoCD and initPassword.isInstall flag
r
Yes, there is a specific nuance when using ArgoCD that changes the usual "set and forget" advice. The ArgoCD Problem ArgoCD does not run 
helm install
or 
helm upgrade
in the traditional sense. Instead, it performs a 
helm template
to generate the manifests and then applies them with 
kubectl
. Because of this, the standard Helm variable 
.Release.IsInstall
is always evaluated as 
true
 by ArgoCD during every single synchronization. In the StarRocks Helm chart, the 
initPassword
job is logic-gated by this variable. If left alone, ArgoCD will try to recreate the "initialize password" Job on every sync, which can lead to: 1. Sync Failures: ArgoCD may report the application as "OutOfSync" or "Degraded" because a Job with the same name already exists or is in a completed state. 2. Unnecessary Noise: Constant logs and pods from the 
kube-starrocks-initpwd
job. The Solution: 
initPassword.isInstall
 To solve this, the StarRocks Helm chart provides an override flag called 
isInstall
 nested under 
initPassword
. 1. Initial Bootstrap: Set 
initPassword.enabled: true
and 
initPassword.isInstall: true
. This allows the Job to be rendered and run for the first time. 2. Post-Bootstrap (ArgoCD Only): Once the cluster is up and the password is set, you should update your GitOps repo to set 
isInstall: false
.
Copy code
yaml
starrocks:
  initPassword:
    enabled: true
    isInstall: false # Set to FALSE for ArgoCD after the first successful sync
    passwordSecret: starrocks-root-pass
Why this is different from the previous answer: While it is technically "safe" (idempotent) for the Job to run again, ArgoCD's resource tracking often gets confused by Jobs that are supposed to be "one-time" but keep appearing in the template output. Summary for your GitOps workflow: * Keep 
enabled: true
(so the configuration remains in your manifests). * Flip 
isInstall
to 
false
after the first successful deployment to keep ArgoCD happy and your "Sync Status" green. References • quick_start/helm.md • deployment/helm.md • administration/user_privs/authentication/native_authentication.md • deployment/post_deployment_setup.md References • quick_start/helm.md • deployment/helm.md • administration/user_privs/authentication/native_authentication.md • deployment/post_deployment_setup.md • ArgoCD Documentation / Community Discussions regarding `.Release.IsInstall` References • quick_start/helm.md • administration/management/FE_configuration.md • faq/Deploy_faq.md
s
@Rocky The PR on githhub mention possibility of setting 
initPassword.image
what do I need to know about it
r
The 
initPassword.image
field was introduced (specifically in PR #454 and Issue #453) to address a specific pain point for users using ArgoCD or other GitOps tools. Here is what you need to know: 1. The Default Behavior If you do not set 
initPassword.image
, the Helm chart defaults to using the same image as your FE nodes (
starrocksFESpec.image
). This makes sense because the FE image already contains the tools and scripts needed to connect to the database and set the password. 2. Why the field was added (The ArgoCD "Bug") In a GitOps workflow, ArgoCD constantly compares the "desired state" in Git with the "live state" in K8S. * If you upgrade your StarRocks version (e.g., from 
3.1.0
to 
3.2.0
), the 
starrocksFESpec.image
changes. * Since the 
init-password
Job inherits that image by default, ArgoCD sees the Job's Pod template has changed. * ArgoCD might then try to re-apply/re-run the Job during the upgrade, even if you already set the password months ago. By setting a specific 
initPassword.image
, you decouple the initialization job from your FE version upgrades. This ensures that the Job's manifest remains static even when you upgrade the rest of the cluster. 3. Recommended Usage If you are using ArgoCD, it is highly recommended to: * Explicitly set the image to a specific version (e.g., use the current FE image version you are deploying). * This prevents the "Sync" status from flipping to "OutOfSync" every time you do a version bump on the actual StarRocks nodes. Example:
Copy code
yaml
starrocks:
  initPassword:
    enabled: true
    # Pin this so FE upgrades don't trigger a Job change
    image: "starrocks/fe-ubuntu:3.2.3" 
    passwordSecret: starrocks-root-pass
4. Summary of Benefits * Decoupling: Upgrading your database version won't trigger "changes" to the old initialization job. * Stability: Avoids unnecessary 
CrashLoopBackOff
or 
Error
logs from old jobs trying to re-run with new images during a cluster maintenance window. * Customization: If you are in a highly restricted environment, you can use a minimal MySQL-client image instead of the heavy StarRocks FE image, provided it can execute the password-reset command. References • starrocks-kubernetes-operator/helm-charts/charts/kube-starrocks/values.yaml • Operator Issue #453: Support setting image for init password job • quick_start/helm.md • deployment/helm.md References • quick_start/helm.md • deployment/helm.md • administration/user_privs/authentication/native_authentication.md • deployment/post_deployment_setup.md • faq/Deploy_faq.md References • quick_start/helm.md • deployment/helm.md
s
@Rocky And what about initPassword.annotations such as mentionned in https://github.com/StarRocks/starrocks-kubernetes-operator/issues/453 - In the end give my the correct incantation for argocd based deployed of initPassword
r
The 
initPassword.annotations
field was added specifically to handle the "Sync-Loop" behavior of ArgoCD. Because ArgoCD uses 
helm template
, it doesn't natively respect Helm's internal installation state, leading to Jobs being treated as persistent resources that are constantly re-evaluated. What to know about 
initPassword.annotations
 By using ArgoCD-specific annotations, you can tell ArgoCD exactly how to manage the lifecycle of the initialization job: 1. 
<http://argocd.argoproj.io/hook|argocd.argoproj.io/hook>: PostSync
: This ensures the Job only runs after ArgoCD has successfully synchronized the FE and BE nodes. This is critical because the Job will fail if it tries to connect to the FE before the FE is actually up and running. 2. 
<http://argocd.argoproj.io/hook-delete-policy|argocd.argoproj.io/hook-delete-policy>: HookSucceeded
: This tells ArgoCD to delete the Job once it finishes successfully. This prevents the Job from sticking around and causing your ArgoCD Application to show a permanent "OutOfSync" or "Progressing" status. ────────── The Correct Incantation for ArgoCD To deploy StarRocks with ArgoCD and manage the root password safely, follow this two-step "incantation." Step 1: Initial Deployment When you first deploy the cluster, use this configuration to trigger the password setup.
Copy code
yaml
starrocks:
  initPassword:
    enabled: true
    isInstall: true  # MUST be true for the first sync
    # Pin the image so future FE upgrades don't change this Job manifest
    image: "starrocks/fe-ubuntu:3.2.3" 
    passwordSecret: starrocks-root-pass
    annotations:
      # Run after the cluster nodes are healthy
      <http://argocd.argoproj.io/hook|argocd.argoproj.io/hook>: PostSync
      # Clean up the Job after it works so ArgoCD stays "Healthy"
      <http://argocd.argoproj.io/hook-delete-policy|argocd.argoproj.io/hook-delete-policy>: HookSucceeded
Step 2: Post-Setup (The "Steady State") Once the first sync is successful and the password is set, update your Git repository to set 
isInstall
to 
false
.
Copy code
yaml
starrocks:
  initPassword:
    enabled: true
    isInstall: false # Flip to FALSE after the first successful sync
    image: "starrocks/fe-ubuntu:3.2.3"
    passwordSecret: starrocks-root-pass
    annotations:
      <http://argocd.argoproj.io/hook|argocd.argoproj.io/hook>: PostSync
      <http://argocd.argoproj.io/hook-delete-policy|argocd.argoproj.io/hook-delete-policy>: HookSucceeded
Why this works: * 
isInstall: false
: Prevents the Helm chart from even rendering the Job manifest in subsequent syncs. This is the ultimate way to stop ArgoCD from trying to run it again. * 
image
: Ensures that if you upgrade StarRocks to version 3.3.0 later, ArgoCD doesn't see a "diff" in the Job (which would exist if it used the default FE image) and try to recreate it. * 
annotations
: Ensures that even if the Job is rendered, ArgoCD handles it as a lifecycle hook rather than a core application resource. References • starrocks-kubernetes-operator/helm-charts/charts/kube-starrocks/values.yaml • Operator Issue #453: Support setting Annotation and image for init password job In ArgoCD • quick_start/helm.md • deployment/helm.md References • quick_start/helm.md • deployment/helm.md • administration/user_privs/authentication/native_authentication.md • deployment/post_deployment_setup.md • faq/Deploy_faq.md References • quick_start/helm.md • deployment/helm.md
s
@Rocky if we disable the job afterward can I just use the default image ? since it won't be rendered anyway the change of container image does not matter ?
r
You are absolutely correct. Your logic is sound for a GitOps workflow. If you set 
isInstall: false
, the Helm template logic completely skips rendering the 
init-password
Job manifest. Since ArgoCD's "Desired State" is based on the output of 
helm template
, the Job disappears from the manifest entirely. Why the default image is fine once disabled: * No Rendering = No Diff: When you upgrade your StarRocks version (e.g., from 
3.1
to 
3.2
), the default value for the Job's image would technically change. However, because the Job manifest isn't being generated at all (due to 
isInstall: false
), ArgoCD has nothing to compare against. It won't see a "diff" because the resource doesn't exist in the new template output. * Orphaned Resource: Once you flip to 
isInstall: false
, the Job is no longer managed by that Helm release. ArgoCD will either ignore the existing (completed) Job or prune it (if you have pruning enabled), but it will not try to "upgrade" it. When would you still want to pin the image? The only reason to set 
initPassword.image
while also using 
isInstall: false
is for future-proofing. If you ever need to reset the password by flipping 
isInstall
back to 
true
, having a pinned image ensures you know exactly which version of the tool is running, rather than accidentally pulling whatever version your FE nodes happen to be on at that moment. Summary: The "Lazy" (but safe) ArgoCD Incantation If you want the simplest configuration that avoids all sync issues: 1. First Sync:
Copy code
yaml
    initPassword:
      enabled: true
      isInstall: true # Renders the Job, uses default FE image
2. Every Sync After:
Copy code
yaml
    initPassword:
      enabled: true
      isInstall: false # Job is gone from manifests; FE image changes won't matter
In this scenario, you do not need to worry about 
initPassword.image
or 
annotations
, because the Job effectively ceases to exist in your GitOps declarations once the password is set. References • starrocks-kubernetes-operator/helm-charts/charts/kube-starrocks/values.yaml • Operator Issue #453: Support setting image for init password job • quick_start/helm.md • deployment/helm.md References • quick_start/helm.md • deployment/helm.md • administration/user_privs/authentication/native_authentication.md • deployment/post_deployment_setup.md • faq/Deploy_faq.md References • quick_start/helm.md • deployment/helm.md • administration/management/FE_configuration.md • faq/Deploy_faq.md
s
@Rocky ArgoCD says my initpwd job is missing
r
If ArgoCD reports the 
initpwd
job as "Missing" (often with a grey icon or a "Missing" status in the resource list), it is usually due to one of two things based on the configuration I gave you earlier: 1. It is working exactly as intended (The "Hook" behavior) If you added the 
<http://argocd.argoproj.io/hook-delete-policy|argocd.argoproj.io/hook-delete-policy>: HookSucceeded
annotation: * ArgoCD runs the job. * The job completes successfully. * The annotation tells Kubernetes to delete the job. * ArgoCD then looks at the cluster, sees the job is gone, and labels it "Missing." Is this a problem? No. In ArgoCD, "Missing" for a Hook is normal. As long as your Application status is "Healthy" (Green heart) and "Synced" (Green check), you can ignore the "Missing" label on that specific resource. It simply means the transient job did its work and cleaned itself up. 2. The "isInstall: false" logic If you switched to 
isInstall: false
, the job is no longer being generated by the Helm template. * If the job still exists on the cluster: ArgoCD will show it as an "Orphaned" or "Extraneous" resource. * If you deleted the job from the cluster: ArgoCD might still show it in the UI as "Missing" if it hasn't fully refreshed its state or if it's still trying to track it as a non-hook resource. ────────── The Final "One-and-Done" Incantation for ArgoCD To stop ArgoCD from complaining and have a fully automated setup (no manual 
isInstall
flipping), use this specific configuration. This treats the job as a Sync Hook that handles its own lifecycle:
Copy code
yaml
starrocks:
  initPassword:
    enabled: true
    isInstall: true # Keep this true so the manifest is always available to ArgoCD
    passwordSecret: starrocks-root-pass
    annotations:
      # This tells ArgoCD: "This is a lifecycle hook, not a permanent part of the app"
      <http://argocd.argoproj.io/hook|argocd.argoproj.io/hook>: PostSync
      # This tells ArgoCD: "Don't count this job toward the health of the application"
      <http://argocd.argoproj.io/hook-delete-policy|argocd.argoproj.io/hook-delete-policy>: HookSucceeded
    # Pin the image to avoid "OutOfSync" during FE version upgrades
    image: "starrocks/fe-ubuntu:3.2.13"
How to clear the "Missing" error right now: If the "Missing" status is preventing a green "Synced" status: 1. Check for Hooks: Ensure the annotations above are in your 
values.yaml
. 2. Prune: In the ArgoCD UI, click Sync and ensure "Prune" is checked. This will tell ArgoCD to stop worrying about resources that are no longer in the Git manifest. 3. Manual Cleanup: If there is a "ghost" job hanging around, delete it manually:
Copy code
bash
    kubectl delete job -l "cluster=kube-starrocks" # or the specific job name
Summary: For ArgoCD, seeing the job as "Missing" is actually the correct state for a successful initialization hook. If your main Application heart is Green, you have succeeded. References • starrocks-kubernetes-operator/helm-charts/charts/kube-starrocks/values.yaml • Operator Issue #453: Support setting Annotation and image for init password job In ArgoCD • quick_start/helm.md • deployment/helm.md References • quick_start/helm.md • deployment/helm.md • administration/user_privs/authentication/native_authentication.md • deployment/post_deployment_setup.md • faq/Deploy_faq.md • administration/management/FE_configuration.md
s
@Rocky Once the FE/BE pods are all booted up properly, is the root password still needed in say pre stop hooks or things like that ? I wonder what happens if we need to rotate the root password while the FE/BE pod are running since the script capture the root password in MYSQL_PWD env. variable that won't get updated.
r
Yes, the root password is absolutely required for the pre-stop hooks in a Kubernetes-managed StarRocks cluster, and rotating it requires careful coordination to avoid breaking pod lifecycle events. 1. Why the password is needed in Pre-Stop Hooks When a BE (Backend) or CN (Compute Node) pod is terminated (e.g., during scale-down or a rolling update), the StarRocks Operator executes a pre-stop script (typically 
/opt/starrocks/be_prestop.sh
or 
cn_prestop.sh
). This script connects to the FE (Frontend) via a MySQL client to perform a graceful decommission (e.g., 
ALTER SYSTEM DECOMMISSION BACKEND "..."
). Without the correct root password, the script cannot authenticate with the FE, and the node will be killed forcefully by Kubernetes after the 
terminationGracePeriodSeconds
expires, potentially leading to tablet inconsistency or query failures. 2. The Problem with Password Rotation As you correctly noted, the password is injected into the pods via the 
MYSQL_PWD
environment variable. In Kubernetes, environment variables are *static*; if you update the underlying Secret, the environment variable inside a running container will not update. If you rotate the root password in the database (via 
SET PASSWORD
) and update the Kubernetes Secret, but do not restart the pods: * Running Pods: Will continue to use the old password stored in their 
MYSQL_PWD
env var. * Pre-Stop Failure: When those pods eventually try to shut down, their pre-stop hooks will attempt to log in with the old password and fail. The FE will reject the connection, and the graceful exit will fail. 3. The Correct "Incantation" for Password Rotation To rotate the password safely without breaking the pre-stop hooks, you must follow this sequence: 1. Update the Database: Log in to StarRocks and change the root password:
Copy code
sql
    SET PASSWORD FOR 'root' = PASSWORD('new_secure_password');
2. Update the K8s Secret: Update the secret referenced by your Helm chart or CRD:
Copy code
bash
    kubectl create secret generic starrocks-root-pass \
      --from-literal=password='new_secure_password' \
      --dry-run=client -o yaml | kubectl apply -f -
3. Trigger a Rolling Restart: You must restart the FE and BE pods so they pick up the new 
MYSQL_PWD
value. * Note: During this rolling restart, the first pods to go down will still have the old password in their env var. Their pre-stop hooks will fail. * Mitigation: To avoid this "first-run" failure during rotation, you can temporarily disable the 
enable_auth_check
in 
fe.conf
, perform the rolling restart, and then re-enable it. However, most users simply accept that the first graceful shutdown during a password rotation may fail and rely on StarRocks' internal replica repair to handle the non-graceful exit. Summary * Is it needed? Yes, for decommissioning nodes during shutdown. * Does 
MYSQL_PWD
update? No. Pods must be restarted to see the new password. * Risk: Pre-stop hooks will fail until the pod is replaced with one containing the new secret value. References • starrocks-kubernetes-operator/doc/change_root_password_howto.md • starrocks-kubernetes-operator/doc/api.md (Pre-stop hook details) • administration/user_privs/authentication/native_authentication.md • deployment/sr_operator.md References • quick_start/helm.md • deployment/helm.md
s
@Rocky Why not use secret has mounted filesystem to avoid stale passwords issue like this.
r
Your intuition is spot on. Using a mounted volume for the secret instead of an environment variable is the standard "Kubernetes way" to handle dynamic credential rotation without pod restarts. However, the current StarRocks Operator (as of v1.x) does not use this approach by default. Here is why it’s not the default and how you can actually implement your idea to solve the rotation problem. 1. Why it's not the default The StarRocks Operator was designed to be simple: it injects the password into the 
MYSQL_PWD
environment variable. * MySQL Client Legacy: The 
mysql
client (used in the pre-stop scripts to decommission nodes) automatically looks for the 
MYSQL_PWD
environment variable. * Static Nature of Env Vars: As you noted, environment variables are set when the container starts and never change. If you rotate the password in the database and the Secret, the 
MYSQL_PWD
inside the running pod becomes stale (incorrect). 2. The "Stale Password" Risk during Pre-Stop If you rotate the password and do not restart the pods, the next time a pod tries to shut down: 1. Kubernetes triggers the 
preStop
hook. 2. The hook runs 
/opt/starrocks/be_prestop.sh
. 3. The script uses the stale 
MYSQL_PWD
to try to log into the FE. 4. The FE rejects the connection. 5. The pod fails to "decommission" itself gracefully and is eventually killed forcefully, which can cause data rebalancing lag or query errors. 3. How to implement your "Mounted Secret" solution You can actually implement the filesystem-mount strategy today by overriding the default behavior in your 
values.yaml
(or StarRocksCluster CRD). Step A: Mount the Secret as a Volume Add the secret as a volume mount to your FE/BE/CN nodes. This ensures the file on disk updates within ~60 seconds of you updating the Kubernetes Secret.
Copy code
yaml
# Example for BE Spec in Helm/CRD
starrocksBeSpec:
  storageVolumes:
    - name: root-password-volume
      mountPath: /etc/starrocks/secrets
      type: secret
      secretName: starrocks-root-pass
Step B: Override the Pre-Stop Hook Instead of letting the Operator use its default script (which relies on the stale env var), you can define a custom 
lifecycle
hook that reads the password from the file right before it runs.
Copy code
yaml
starrocksBeSpec:
  lifecycle:
    preStop:
      exec:
        command:
          - /bin/sh
          - -c
          - |
            # Dynamically load the password from the mounted volume
            export MYSQL_PWD=$(cat /etc/starrocks/secrets/password)
            # Run the original pre-stop script with the updated password
            /opt/starrocks/be_prestop.sh
Summary of Benefits * No Stale Passwords: The 
cat
command in the hook reads the "live" version of the secret from the filesystem every time it runs. * Zero-Downtime Rotation: You can change the root password in StarRocks and update the K8s Secret without needing to roll the entire cluster just to update environment variables. * Reliable Graceful Shutdown: Your pre-stop hooks will always have the correct credentials to talk to the FE and decommission the node safely. Is there a catch? The only minor catch is that ArgoCD (if you use it) might see your manual 
lifecycle
override and the Operator's default hook as a conflict unless you ensure the Operator isn't fighting to put its default hook back. Most modern versions of the StarRocks Operator allow you to override these fields in the Spec without conflict. References • starrocks-kubernetes-operator/doc/change_root_password_howto.md • starrocks-kubernetes-operator/doc/api.md (Specifying lifecycle hooks) • deployment/helm.md
Open in Slack
PreviousNext
Powered by