@Rocky - starrocks 3.5.20 shared-data mode. Showing worker group 0 does not have alive workers on startup. What's going on and how it can be fixed?

This error occurs when the StarRocks Frontend (FE) cannot find any active Compute Nodes (CN) within the specified worker group (in this case, group 0, which typically corresponds to the

default_warehouse

or the system's default resource group). In shared-data mode, the FE schedules query tasks to "workers" (CNs) grouped into warehouses. If the FE reports that a worker group has no alive workers, it means the scheduler has no available compute resources to execute your request. Common Causes and Fixes 1. Compute Nodes (CN) are not registered or alive The most common cause is that the CN processes haven't successfully joined the cluster or have crashed. * Check Status: Run the following SQL command to check if any CNs are listed and if they are marked as `Alive: true`:

sql
    SHOW COMPUTE NODES;
    -- or if using older versions
    SHOW BACKENDS;

* Fix: If the list is empty, you must add your CNs to the cluster:

sql
    ALTER SYSTEM ADD COMPUTE NODE "cn_host:heartbeat_port";

If they are listed but

Alive

is

false

, check the

cn.out

and

<http://cn.INFO|cn.INFO>

logs on the CN machines for startup errors. 2. Storage/Object Store Connection Issues In shared-data mode, CNs must be able to access the remote object storage (S3, MinIO, etc.) to function. If a CN cannot initialize its connection to the storage defined in the FE's

run_config

or the warehouse configuration, it may fail to report as a healthy worker. * Check Logs: Look for "failed to init object storage" or "heartbeat" errors in

<http://cn.INFO|cn.INFO>

. * Fix: Ensure the CN has network access to the storage endpoint and that the credentials/endpoint configured in the cluster are correct. 3. Warehouse/Worker Group Mismatch If you are using the Multi-Warehouse feature (available in StarRocks 3.x+), the query might be directed to a warehouse that has no nodes assigned or is suspended. * Check Warehouses:

sql
    SHOW WAREHOUSES;

* Fix: Ensure the warehouse you are using has at least one CN assigned. If the default warehouse is empty, you can add nodes to it or ensure your CNs are not assigned to a different, specific warehouse. 4. Heartbeat and Network Issues The FE and CN communicate via heartbeat ports (default

9050

). If there is a firewall or network partition, the FE will mark the CN as "not alive." * Fix: Verify that the FE can reach the CN's

heartbeat_port

and the CN can reach the FE's

query_port

and

edit_log_port

. 5. StarOS Metadata Mismatch (Advanced) Shared-data mode relies on a metadata service (internal StarOS). If the CNs were restarted with a different

starlet

configuration or if the metadata on the FE was reset/wiped, the mapping of worker group 0 might be broken. * Fix: Check

fe.log

for any "StarOS" or "LakeService" exceptions. If this is a fresh deployment, ensure the

cloud_native_storage_type

and related storage parameters are consistent across all FE nodes. Recommended first step: Run

SHOW COMPUTE NODES

and check the

ErrMsg

column for any specific error reasons reported by the FE. References • faq/shared_data_faq.md • quick_start/shared-data.md • deployment/shared_data/feature-support-shared-data.md • deployment/shared_data/s3.md

@Rocky - FE cluster is starting up from scratch and is set up to not start CN nodes before at least one FE is up. Two sit in follower state, one in init and the one in init complains about workers not being there

In StarRocks 3.5.20 shared-data mode, seeing "worker group 0 does not have alive workers" during a fresh startup is a common "chicken-and-egg" situation that often occurs when trying to stabilize the Frontend (FE) cluster before starting Compute Nodes (CN). 1. What's going on? In shared-data mode, StarRocks uses a Warehouse architecture. The

default_warehouse

(id 0) and its

default_worker_group

(id 0) are automatically managed by the FE. * Background Tasks: Even before you run any queries, the FE starts background services like the

LakeService

,

CompactionScheduler

, and

WarehouseManager

. These services periodically check for available compute resources to perform internal tasks (such as metadata maintenance or checking storage accessibility). * The Error: Since you haven't started any CN nodes yet, the FE's scheduler correctly reports that worker group 0 has no alive workers. In many cases, this is background noise and not necessarily what is blocking your third FE from moving out of the

INIT

state. * The

INIT

State: An FE in

INIT

means it has not yet completed its

GlobalStateMgr

initialization or has not successfully synchronized its metadata with the Leader. 2. Is it a blocker? Usually, the "no alive workers" log is a symptom, not the cause of an FE being stuck in

INIT

. However, in some 3.x versions, if the FE is trying to perform a mandatory system initialization task (like creating system tables or validating the storage volume) that requires a compute resource, it might "hang" or retry until a CN is available. 3. How to fix and move forward Follow these steps to resolve the state and finish the cluster bootstrap: A. Verify the FE Quorum If you have 3 FEs and 2 are in

FOLLOWER

state, a Leader should have been elected (one of the followers will show

IsLeader: true

in

SHOW FRONTENDS

). * Check the

fe.log

of the node in

INIT

for other errors. Specifically, look for RPC timeouts, Connection refused, or BDBJE errors. * Ensure

priority_networks

is configured in

fe.conf

on all nodes to prevent them from trying to communicate over the wrong network interface. B. Start and Register at least one CN StarRocks shared-data mode requires at least one CN to be fully functional. You do not need to wait for all FEs to be "Ready" before starting a CN. Once the Leader FE is up (even if the other two are still syncing): 1. Start one CN node (

./bin/start_cn.sh --daemon

). 2. Connect to the Leader FE via MySQL and add the CN:

sql
    ALTER SYSTEM ADD COMPUTE NODE "cn_host:heartbeat_service_port";

3. Check if the CN is alive:

SHOW COMPUTE NODES;

. Once a CN is registered and

Alive: true

, the "worker group 0" error should stop, and any background initialization tasks that were waiting for a worker will complete, which may allow the

INIT

FE to finish its startup. C. Manual Warehouse Check (If the error persists after CN is added) If the CN is added but the error continues, check the warehouse status:

sql
SHOW WAREHOUSES;
-- If no warehouse exists, the default should be created automatically, 
-- but you can try to trigger a refresh by restarting the Leader FE.

Summary Recommendation Don't wait for the third FE to leave

INIT

before starting your CNs. Start your CN nodes now and register them with the FE cluster. This often clears the background errors and allows the remaining FE to synchronize and become

Ready

. If the

INIT

node remains stuck after adding a CN, check its

fe.log

specifically for "failed to get leader" or "catch up" messages, as that points to a network or metadata consistency issue between the FEs. References • administration/Meta_recovery.md • quick_start/shared-data.md

@Rocky - The only abnormality I see on the INIT node is this one: @Rocky I only see this: 2026-10-20 102601.240-04:00 WARN (Finalizer|3) [ResolvingFileIO.finalize():266] Unclosed ResolvingFileIO instance created by: org.apache.iceberg.io.ResolvingFileIO.<init>(ResolvingFileIO.java:84) com.starrocks.connector.iceberg.io.IcebergCachingFileIO.initialize(IcebergCachingFileIO.java:115) org.apache.iceberg.CatalogUtil.loadFileIO(CatalogUtil.java:402) org.apache.iceberg.CatalogUtil.loadFileIO(CatalogUtil.java:349) org.apache.iceberg.hive.HiveCatalog.initialize(HiveCatalog.java:123) org.apache.iceberg.CatalogUtil.loadCatalog(CatalogUtil.java:277) com.starrocks.connector.iceberg.hive.IcebergHiveCatalog.<init>(IcebergHiveCatalog.java:97) com.starrocks.connector.iceberg.IcebergConnector.buildIcebergNativeCatalog(IcebergConnector.java:80) com.starrocks.connector.iceberg.IcebergConnector.getNativeCatalog(IcebergConnector.java:104) com.starrocks.connector.iceberg.IcebergConnector.getMetadata(IcebergConnector.java:96) com.starrocks.connector.LazyConnector.getMetadata(LazyConnector.java:42) com.starrocks.connector.CatalogConnector.getMetadata(CatalogConnector.java:45) com.starrocks.server.MetadataMgr.getOptionalMetadata(MetadataMgr.java:202) com.starrocks.server.MetadataMgr.getOptionalMetadata(MetadataMgr.java:176) com.starrocks.server.MetadataMgr.getTable(MetadataMgr.java:501) com.starrocks.server.MetadataMgr.getTable(MetadataMgr.java:543) com.starrocks.server.MetadataMgr.getTableWithIdentifier(MetadataMgr.java:548) com.starrocks.server.MetadataMgr.getTableChecked(MetadataMgr.java:562) com.starrocks.sql.optimizer.rule.transformation.materialization.MvUtils.getTableChecked(MvUtils.java:1461) com.starrocks.catalog.MaterializedView.lambda$getBaseTableInfosWithoutView$0(MaterializedView.java:774) java.base/java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:178) java.base/java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1625) java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509) java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499) java.base/java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:921) java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) java.base/java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:682) com.starrocks.catalog.MaterializedView.getBaseTableInfosWithoutView(MaterializedView.java:775) com.starrocks.catalog.MaterializedView.analyzePartitionExprs(MaterializedView.java:2228) com.starrocks.catalog.MaterializedView.onReloadImplHeavy(MaterializedView.java:1352) com.starrocks.catalog.MaterializedView.checkAndSetActive(MaterializedView.java:1291) com.starrocks.catalog.MaterializedView.onReload(MaterializedView.java:1249) com.starrocks.catalog.MaterializedView.onReload(MaterializedView.java:1220) com.starrocks.catalog.MaterializedView.checkIsActiveOnLoadBlocking(MaterializedView.java:1436) com.starrocks.catalog.MaterializedView.lambda$checkAndSetActive$3(MaterializedView.java:1279) java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1768) java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) java.base/java.lang.Thread.run(Thread.java:840 And on the followers this is what it reports: 2026-10-20 094825.803-04:00 WARN (Finalizer|3) [ResolvingFileIO.finalize():266] Unclosed ResolvingFileIO instance created by: org.apache.iceberg.io.ResolvingFileIO.<init>(ResolvingFileIO.java:84) com.starrocks.connector.iceberg.io.IcebergCachingFileIO.initialize(IcebergCachingFileIO.java:115) org.apache.iceberg.CatalogUtil.loadFileIO(CatalogUtil.java:402) org.apache.iceberg.CatalogUtil.loadFileIO(CatalogUtil.java:349) org.apache.iceberg.hive.HiveCatalog.initialize(HiveCatalog.java:123) org.apache.iceberg.CatalogUtil.loadCatalog(CatalogUtil.java:277) com.starrocks.connector.iceberg.hive.IcebergHiveCatalog.<init>(IcebergHiveCatalog.java:97) com.starrocks.connector.iceberg.IcebergConnector.buildIcebergNativeCatalog(IcebergConnector.java:80) com.starrocks.connector.iceberg.IcebergConnector.getNativeCatalog(IcebergConnector.java:104) com.starrocks.connector.iceberg.IcebergConnector.getMetadata(IcebergConnector.java:96) com.starrocks.connector.LazyConnector.getMetadata(LazyConnector.java:42) com.starrocks.connector.CatalogConnector.getMetadata(CatalogConnector.java:45) com.starrocks.server.MetadataMgr.getOptionalMetadata(MetadataMgr.java:202) com.starrocks.server.MetadataMgr.getOptionalMetadata(MetadataMgr.java:176) com.starrocks.server.MetadataMgr.getTable(MetadataMgr.java:501) com.starrocks.server.MetadataMgr.getTable(MetadataMgr.java:543) com.starrocks.server.MetadataMgr.getTableWithIdentifier(MetadataMgr.java:548) com.starrocks.server.MetadataMgr.getTableChecked(MetadataMgr.java:562) com.starrocks.sql.optimizer.rule.transformation.materialization.MvUtils.getTableChecked(MvUtils.java:1461) com.starrocks.catalog.MaterializedView.lambda$getBaseTableInfosWithoutView$0(MaterializedView.java:774) java.base/java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:178) java.base/java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1625) java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509) java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499) java.base/java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:921) java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) java.base/java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:682) com.starrocks.catalog.MaterializedView.getBaseTableInfosWithoutView(MaterializedView.java:775) com.starrocks.catalog.MaterializedView.analyzePartitionExprs(MaterializedView.java:2228) com.starrocks.catalog.MaterializedView.onReloadImplHeavy(MaterializedView.java:1352) com.starrocks.catalog.MaterializedView.checkAndSetActive(MaterializedView.java:1291) com.starrocks.catalog.MaterializedView.onReload(MaterializedView.java:1249) com.starrocks.catalog.MaterializedView.onReload(MaterializedView.java:1220) com.starrocks.catalog.MaterializedView.checkIsActiveOnLoadBlocking(MaterializedView.java:1436)

The error and stack traces you are seeing indicate that your FE cluster is currently stuck in the Metadata Reload phase, specifically trying to initialize Materialized Views (MVs) that reference external Iceberg tables. Here is the breakdown of what is happening and how to fix it: 1. Why is the FE stuck in

INIT

? When a StarRocks FE starts up (even a new node joining an existing cluster), it must reload the global metadata. Part of this process involves: *

onReload

/

checkAndSetActive

: The FE iterates through all existing Materialized Views to verify their state. * Iceberg Metadata Fetch: The stack trace shows the FE is reaching out to an external Iceberg catalog (via

IcebergHiveCatalog

and

ResolvingFileIO

) to check the base tables of those MVs. * The Block: If the FE cannot reach the Iceberg catalog (Hive MetaStore or S3) or if it's waiting for a response that never comes, the thread blocks. This prevents the FE from completing its initialization and moving from

INIT

to a healthy

FOLLOWER

or

LEADER

state. 2. The "Worker Group 0" Connection In shared-data mode, worker group 0 (the

default_worker_group

of the

default_warehouse

) is the primary compute resource for system tasks. * Certain "heavy" metadata operations or background maintenance tasks in shared-data mode require an active worker node (CN) to be available to handle storage-related checks or internal query fragments. * The warning

worker group 0 does not have alive workers

confirms that the FE has no compute resources to delegate these tasks to. 3. The "Unclosed ResolvingFileIO" Warning This is a side effect. It means the Iceberg connection (FileIO) was opened to fetch metadata but was never properly closed—likely because the operation timed out or the thread was interrupted because the FE is struggling to stabilize. ────────── How to Fix Step 1: Start and Register your CN nodes In shared-data mode, you should not wait for all FEs to be in a perfect state before starting CNs. The cluster lifecycle in 3.x shared-data mode expects compute resources to be available almost immediately for background initialization. 1. Start your CN nodes. 2. If you haven't already, register them with the FE (use the FE that is currently the Leader—check

SHOW FRONTENDS

on one of the

FOLLOWER

nodes to see who is

IsLeader: true

):

sql
    ALTER SYSTEM ADD COMPUTE NODE "cn_host:9050";

Once the FE sees "Alive" workers, it can successfully initialize the

default_warehouse

, which often clears the block on background metadata tasks. Step 2: Check External Connectivity The stack trace shows a failure to communicate with an Iceberg catalog. Ensure that: * The FE nodes have network access to the Hive MetaStore or Glue endpoint. * The FE nodes have the correct S3/Object Store credentials (via

fe.conf

or instance profiles) to read the Iceberg metadata files. * If you are using a shared metadata volume from a previous installation, the FE is trying to reach catalogs that might no longer exist or are unreachable. Step 3: Increase Metadata Loading Timeouts (Optional) If you have a very large number of Materialized Views, the reload can take time. You can check

fe.log

for any "timeout" messages related to

GlobalStateMgr

. However, adding a CN is the most likely fix for the "worker group" error specifically. Summary: The "worker group 0" error is a signal that your FEs are trying to do work (MV initialization) but have no compute nodes to help. Start your CNs now and the

INIT

node should eventually catch up and transition to

FOLLOWER

. References • using_starrocks/async_mv/troubleshooting_asynchronous_materialized_views.md • knowledge/trouble_shooting/fe_deadlock_problems.md

@Rocky - what's the name of the property in the values.yaml file that will prevent CNs from loading until FEs are loaded?

In StarRocks deployments using the StarRocks Operator (which is the standard way to deploy via Helm using the

kube-starrocks

chart), the behavior you are describing is actually the built-in default logic of the Operator's reconciliation loop. There isn't a single "toggle" property like `waitForFE: true`; instead, the StarRocks Operator is designed as a "Level 2" operator that manages component dependencies automatically. 1. The "Property" (Operator Logic) The Operator's internal logic prevents the creation or scaling of

starRocksCnSpec

(Compute Nodes) until the

starRocksFESpec

(Frontends) have reached a Ready state. * In your case, because you have one FE stuck in

INIT

and the others are likely not yet reporting as fully

Ready

to the Kubernetes readiness probe, the Operator is holding back the CN nodes. 2. The Deadlock (The "Chicken-and-Egg" Problem) You have encountered a common dependency deadlock in shared-data mode (especially with version 3.x and external catalogs like Iceberg): 1. Operator Logic: "I won't start the CNs until the FEs are Ready." 2. FE Status: The FE stays in

INIT

because it is trying to reload metadata (like checking Materialized Views or external Iceberg tables). 3. Shared-Data Requirement: In shared-data mode, the FE often needs an active worker (CN) in

worker group 0

to execute the internal tasks required to finish its own initialization. How to Fix It To break this deadlock, you need to allow the CNs to start even if the FE is not yet "Ready." You can try the following: * Check Readiness Probes: If your FEs are actually running but just not "Ready" in Kubernetes terms, the Operator will wait indefinitely. You might need to temporarily relax the

readinessProbe

in your

values.yaml

for the FE so that the Operator thinks they are ready and triggers the CN deployment. * Manual Intervention: If you are using the Operator, check the

StarRocksCluster

resource status (

kubectl get starrockscluster

). If it is stuck in

Reconciling

, it's because of the FE. * Use

initContainers

(The Manual Way): If you manually added a wait script in the

values.yaml

under

starrocksCnSpec.initContainers

, that is what's blocking the CN. It usually looks like this:

yaml
    starrocksCnSpec:
      initContainers:
      • name: wait-for-fe
        image: busybox:1.28
        command: ['sh', '-c', 'until nslookup <fe-service-name>; do sleep 2; done']

If this exists in your

values.yaml

, remove or comment it out to let the CN start and assist the FE. Summary: The "property" is the Operator's built-in reconciliation sequence. Because your FE is stuck in

INIT

(waiting for workers to help load Iceberg/MV metadata), the Operator is refusing to start the workers. You should check if you can manually start at least one CN or relax the FE's readiness requirements in the Helm chart to trigger the Operator to proceed. References • quick_start/shared-data.md • deployment/deploy_manually.md • administration/Meta_recovery.md • quick_start/routine-load.md

@Rocky - even when enabledCN: false is set in values.yaml the cluster upon restart still complains about no CNs.

In StarRocks shared-data mode, the setting

enabledCN: false

in your

values.yaml

only prevents the Kubernetes Operator from deploying the CN Pods. It does not tell the StarRocks engine to ignore the lack of compute nodes. Because you are in shared-data mode, the FE cluster is architecturally dependent on having workers (CNs) to perform background system tasks. This is why you are seeing the "no alive workers" error even with the setting disabled. 1. Why

enabledCN: false

doesn't stop the error In shared-data mode (Cloud Native), the Frontend (FE) delegates nearly all data-related tasks to the Compute Nodes (CN). Upon startup, the FE initiates several background services: * Warehouse Manager: Automatically tries to initialize

worker group 0

in the

default_warehouse

. If no CNs are registered or alive, it will log warnings/errors indefinitely. * Metadata Reload (The Block): Your previous logs showed the FE stuck in

MaterializedView.onReload

. In StarRocks 3.x, if an MV references an external table (like Iceberg), the FE may need to dispatch a metadata-fetch task. If there are no CNs, this task cannot be scheduled, causing the FE thread to block or retry, keeping the node in the

INIT

state. 2. The Deadlock (Chicken-and-Egg) You are likely experiencing a common deadlock with the StarRocks Operator: 1. Operator Logic: It waits for the FE cluster to be "Ready" (pass Kubernetes readiness probes) before it starts the CN nodes. 2. FE State: The FE stays in

INIT

(Not Ready) because it is waiting for a worker to help it finish loading Materialized View metadata. 3. Result: The CNs never start because the FEs aren't ready, and the FEs never become ready because the CNs aren't there to help. 3. How to resolve this To break this cycle, you must allow at least one CN to start and join the cluster so the FEs can complete their initialization. Option A: Relax the FE Readiness Probe (Recommended) Modify your

values.yaml

to make the FE's readiness probe less strict. This allows the Operator to see the FEs as "Ready" (even if they are still in

INIT

internally), which triggers the deployment of the CNs.

yaml
starrocksFESpec:
  readinessProbe:
    failureThreshold: 60  # Increase significantly
    initialDelaySeconds: 10
    periodSeconds: 10

Option B: Manually Force CN Deployment If you are comfortable with

kubectl

, you can check if the

StarRocksCluster

(SRC) resource has a

starRocksCnSpec

. If the Operator is refusing to create the CN Deployment because the FE isn't ready, you may need to temporarily set

enabledCN: true

and check the Operator logs (

kubectl logs -n <namespace> <operator-pod-name>

) to see why it isn't reconciling the CNs. Option C: Bypass MV Reload (Advanced) If you believe the Materialized Views are causing a terminal block, you can try to start the FE with background MV reloads disabled by adding this to your

fe.conf

(via

starrocksFESpec.config

in

values.yaml

):

properties
# This might allow the FE to finish INIT without waiting for external catalogs
enable_background_refresh_materialized_view = false

Note: Use this only as a temporary measure to get the cluster up, then re-enable it once CNs are healthy. Summary Shared-data StarRocks cannot function without CNs. Setting

enabledCN: false

effectively breaks the cluster. You should: 1. Set

enabledCN: true

. 2. Ensure your FE readiness probes aren't so strict that they prevent the CNs from ever being created. 3. Once the first CN is up and registers with the Leader FE, the

INIT

node should transition to

FOLLOWER

shortly after. References • deployment/deploy_manually.md • deployment/sr_operator.md • deployment/upgrade.md • deployment/shared_data/s3.md

@Rocky - what is the correct name of the enable_background_refresh_materialized_view property?

References • administration/management/FE_configuration.md • deployment/deploy_manually.md

@Rocky - how do I force mark one of the FEs as ready?

In the context of the StarRocks Operator, you cannot "force" a pod to be ready by clicking a button or setting a single flag. However, you can bypass the health check that is causing the FE to be marked as "Not Ready." Since the Operator waits for the FE to be

Ready

before starting CNs, and the FE won't become

Ready

until it sees a CN (the "chicken-and-egg" deadlock), you should relax the Readiness Probe. 1. The "Bypass" Method (Recommended) You can modify your

values.yaml

to replace the standard health check with a command that always returns success. This "tricks" Kubernetes and the Operator into thinking the FE is ready, which triggers the deployment of the CNs. In your

values.yaml

, override the

readinessProbe

for the FE:

yaml
starrocksFESpec:
  readinessProbe:
    exec:
      command:
        - /bin/sh
        - -c
        - "exit 0"
    initialDelaySeconds: 5
    periodSeconds: 5

2. The "Patience" Method If you don't want to bypass the check entirely, you can significantly increase the

failureThreshold

and

initialDelaySeconds

. This gives the FE a very long time to "eventually" become ready without Kubernetes killing it or the Operator giving up, though it won't solve the deadlock if the FE truly requires a CN to finish its

INIT

phase.

yaml
starrocksFESpec:
  readinessProbe:
    httpGet:
      path: /api/health
      port: 8030
    failureThreshold: 300  # Allow up to 50 minutes (300 * 10s)
    periodSeconds: 10

3. Why this works * The Operator: It monitors the

Ready

status of the FE Pods. Once it sees

1/1 Ready

, its reconciliation logic moves to the next step: creating/scaling the CNs. * The FE: Once the CN starts and registers with the FE, the FE will finally be able to complete its background metadata tasks (like reloading Materialized Views) and transition from

INIT

to

FOLLOWER

or

LEADER

. Important: After the cluster successfully starts and the CNs are running, you should revert the readiness probe to the standard settings so that Kubernetes can accurately detect and restart the FE if it actually crashes in the future. Verification After applying the change, run:

bash
kubectl get pods -n <your-namespace>

You should see the FE pods show

1/1 READY

immediately. Shortly after, you should see the Operator begin to create the CN pods. References • quick_start/helm.md • deployment/sr_operator.md • deployment/deploy_manually.md • faq/Deploy_faq.md References • faq/Deploy_faq.md • administration/management/FE_configuration.md • deployment/deploy_manually.md • deployment/upgrade.md • deployment/sr_operator.md • faq/operation_maintenance_faq.md • administration/management/graceful_exit.md

@Rocky - in values.yaml - what specifically do I need to change for FE settings block so that cluster considers FEs ready up and running without them actually be ready

To force the StarRocks Operator to consider the FE pods "Ready" (even if they are internally stuck in the

INIT

state), you need to override the default Readiness Probe in your

values.yaml

. This is the standard way to break the "chicken-and-egg" deadlock where the FE waits for a CN to finish its startup, but the Operator won't start the CN until the FE is "Ready." The Change for

values.yaml

Add or modify the

readinessProbe

block within

starrocksFESpec

. Using an

exec

probe that returns

exit 0

will force Kubernetes to mark the pod as healthy immediately.

yaml
starrocks:
  starrocksFESpec:
    # This block forces the FE to be marked as Ready immediately
    readinessProbe:
      exec:
        command:
          - /bin/sh
          - -c
          - "exit 0"
      initialDelaySeconds: 5
      periodSeconds: 5

Why this specific change? * The Problem: The default probe usually checks

httpGet

on port

8030/api/health

. If the FE is stuck in

INIT

(Materialized View reloading, etc.), this endpoint returns a failure. * The Result: By switching to

exec: command: ["exit 0"]

, the kubelet always receives a "success" signal. * The Outcome: The Operator sees the FE pods as

1/1 Ready

, which satisfies its requirement to move forward and finally deploy your CN/BE pods. Alternative: The "High Threshold" approach If you don't want to bypass the check entirely but want to stop the Operator from restarting the FE or blocking the CN, you can increase the failure threshold to a very high number:

yaml
starrocks:
  starrocksFESpec:
    readinessProbe:
      httpGet:
        path: /api/health
        port: 8030
      failureThreshold: 100 # Allow ~15-20 minutes of "Not Ready" before giving up
      periodSeconds: 10

Recommendation: Use the

exit 0

method to get the cluster up for the first time. Once the CNs are registered and the FE finally transitions to

LEADER

or

FOLLOWER

, you should remove this override to restore proper health monitoring. References • quick_start/helm.md • deployment/sr_operator.md • faq/Deploy_faq.md • administration/management/FE_configuration.md References • quick_start/helm.md • administration/management/FE_configuration.md

@Rocky - Readiness: http-get http//8030/api/health delay=0s timeout=1s period=5s #success=1 #failure=3 even after I try starrocks: starrocksFESpec: # This block forces the FE to be marked as Ready immediately readinessProbe: exec: command: - /bin/sh - -c - "exit 0" initialDelaySeconds: 5 periodSeconds: 5

Based on your output, it appears the StarRocks Operator is ignoring your

readinessProbe

configuration and falling back to its hardcoded default (

http-get http://:8030/api/health

). This usually happens for one of two reasons: the version of the operator you are using doesn't support full probe overrides, or the structure/indentation in your

values.yaml

is slightly off for the specific Helm chart you are using. 1. Verify the Indentation and Key Depending on whether you are using the kube-starrocks chart (which includes the operator) or the standalone starrocks cluster chart, the key might differ. Ensure your

values.yaml

follows this exact structure (note the nesting): For the

kube-starrocks

chart:

yaml
starrocks:
  starrocksFESpec:
    readinessProbe:
      exec:
        command:
          - /bin/sh
          - -c
          - "exit 0"
      initialDelaySeconds: 5
      periodSeconds: 5

2. Check for Operator Field Support In some versions of the StarRocks Operator, the

readinessProbe

field is not fully exposed as a block. Instead, it uses specific fields to control the probe behavior. If the

exec

block continues to be ignored, try using the specific Failure Seconds fields:

yaml
starrocks:
  starrocksFESpec:
    # Some operator versions use these specific fields instead of a full probe block
    readinessProbeFailureSeconds: 3600  # Set to 1 hour to prevent "Not Ready" status from affecting the cluster
    livenessProbeFailureSeconds: 3600

3. Check the StarRocksCluster CR To see if your change actually made it into the Kubernetes cluster configuration, check the Custom Resource (CR) directly:

bash
kubectl get starrockscluster -n <your-namespace> -o yaml

* If the

readinessProbe

is missing from the CR output: Your

values.yaml

structure is incorrect or the Helm chart version is too old to pass that field to the operator. * If the

readinessProbe

is present in the CR but the Pod still shows `http-get`: The version of the StarRocks Operator you have installed does not support custom probes and is hardcoding the health check. Recommended Version Customizing probes was improved in Operator v1.8.7+. If you are on an older version, the operator will always default to the

http-get

check on port 8030. You may need to upgrade the operator or use the

readinessProbeFailureSeconds

workaround. References • quick_start/helm.md • deployment/sr_operator.md • faq/Deploy_faq.md • administration/management/FE_configuration.md References • quick_start/helm.md • deployment/deploy_manually.md • administration/management/monitoring/Monitor_and_Alert.md

@Rocky - I've set the readinessProbeseconds to 0. Operator version 1.11.7. Still see the http probe on readiness

To "force" an FE to be marked as ready in StarRocks Operator v1.11.7, you should use the specific field designed for this purpose:

readinessProbeFailureSeconds

. The reason your previous attempt with the

exec

block didn't work is that the StarRocks Operator CRD (Custom Resource Definition) abstracts the readiness probe. It does not allow a full Kubernetes

readinessProbe

object override (like

exec

or

httpGet

); instead, it generates the probe for you based on specific fields. The Correct Settings In your

values.yaml

, ensure you are using the correct casing and field name. Setting this to

0

tells the Operator to disable the readiness probe in the Pod specification entirely.

yaml
starrocks:
  starrocksFESpec:
    # Use exact casing: readinessProbeFailureSeconds
    # Setting this to 0 removes the probe from the Pod spec
    readinessProbeFailureSeconds: 0
    # Optional: You may want to do the same for liveness if it's also failing
    livenessProbeFailureSeconds: 0

Why your previous attempts might have failed: 1. Casing: You mentioned trying

readinessProbeseconds

. Kubernetes and the StarRocks Operator are case-sensitive. If the field is not exactly

readinessProbeFailureSeconds

, it will be ignored, and the operator will fall back to its hardcoded default (

http-get

on port 8030). 2. Object Structure: Using a

readinessProbe:

block with

exec:

is not supported by the StarRocksCluster CRD. The operator only looks for the

...FailureSeconds

fields to configure health checks. How to verify it worked: 1. Update the cluster:

bash
    helm upgrade starrocks starrocks/kube-starrocks -f values.yaml

2. Check the Pod Spec: Wait for the FE pods to restart (or delete them to force a restart), then check the description:

bash
    kubectl describe pod <fe-pod-name>

If successful, the

Readiness:

line will be completely missing from the container description. 3. Wait for the Operator: Once the pod is running (even if the FE is still "initializing" internally), the absence of a failing readiness probe means Kubernetes will report the pod as

1/1 READY

. This will signal the Operator to start deploying your CN or BE nodes. Note: Once your cluster is fully initialized and CNs are registered, it is highly recommended to set

readinessProbeFailureSeconds

back to a reasonable value (e.g.,

300

) to ensure the cluster can properly recover from future failures. References • deployment/sr_operator.md • quick_start/helm.md • faq/Deploy_faq.md • administration/management/FE_configuration.md References • quick_start/helm.md • deployment/sr_operator.md

@Rocky - starrocks cn statefulset not found in kubernetes operator logs

In the StarRocks Operator (v1.11.7), the error message "cn statefulset not found" in the logs typically occurs because the Operator's reconciliation loop is trying to manage the CN component, but the actual Kubernetes StatefulSet resource has not been created yet. This is usually caused by one of the following three scenarios: 1. The FE is "Ready" but not "Functional" Even though you forced the FE pods to be

1/1 Ready

(by bypassing the readiness probe), the StarRocks Operator often waits for the FE to be functionally healthy before it creates the CN or BE StatefulSets. * The Check: The Operator attempts to connect to the FE's internal API to verify if a Leader has been elected. * The Result: If the FE is still stuck in an

INIT

state (e.g., waiting for metadata or a CN to join), the Operator stays in a loop. It logs "not found" because it hasn't reached the step where it actually

Creates

the StatefulSet, but the reconciliation logic is already looking for it to see if it needs updates. Solution: Check the FE logs (

kubectl logs <fe-pod-name>

) to see if it says "wait for leader election" or "image is not ready." If the FE is not yet a Leader, the CN will not be deployed. 2. Incorrect YAML Nesting If the

starRocksCnSpec

is not at the correct indentation level in your

values.yaml

, the Operator will ignore the CN configuration entirely, resulting in no StatefulSet being created. Ensure your

values.yaml

looks like this:

yaml
starrocks:
  starrocksFESpec:
    # ... your FE config ...
  starrocksCnSpec:  # <--- MUST be at the same level as starrocksFESpec
    enabled: true   # Some charts require this explicit flag
    replicas: 3
    image:
      repository: starrocks/cn-ubuntu
      tag: 3.1-latest

3. Missing Storage/Object Store Configuration (Shared-Data) CN (Compute Nodes) are primarily used in Shared-Data (Cloud-Native) mode. If you are deploying CNs but have not provided the mandatory object storage configuration (S3/GCS/Azure) in the

starRocksFeSpec

, the Operator may fail to initialize the FE properly, which in turn prevents the CN StatefulSet from being created. How to Diagnose Run the following command to see what the Operator actually "sees" in your configuration:

bash
kubectl get starrockscluster -o yaml

* Look for

spec.starRocksCnSpec

: If this section is missing from the output, the Operator doesn't know you want CNs (check your

values.yaml

indentation). * Look for

status

: Check the

cnStatus

or

feStatus

fields at the bottom of the output. If the

feStatus

does not say

running

, the Operator will not create the CN. Summary The "not found" log is often a transient warning. The Operator says "I can't find the CN StatefulSet (so I can't update it)," but it won't create it until the FE reports that it is ready to accept compute nodes. Ensure at least one FE has successfully elected itself as Leader by checking

kubectl logs <fe-pod> | grep "master"

. References • deployment/sr_operator.md • quick_start/helm.md • administration/management/logs.md • deployment/deploy_manually.md References • deployment/deploy_manually.md • deployment/sr_operator.md • deployment/environment_configurations.md