@Rocky Starrcosk fe pod create huge error and cant init to running status. what kind of reason makes this problems and Error Log? (version 3.5.10) • BaseMetastoreTableOperations.refreshFromMetadataLocation():190 Refreshing table metadata from new version • Error ConnectorHistorgramColumnStatsCacheLoader.lambda@asyncLoadAll$1():105 com.starrocks.common.ErrorReportException: Warehouse id: 0 • WARN CachedStatisticStorage.lambda@getConnectorTableStatistics$7():224 Get connector table column statistics filed exception

Based on your error logs and version (3.5.10), the FE pod failure is caused by a race condition or an initialization failure in the Warehouse Management system, specifically relating to the

default_warehouse

(which has an ID of

0

). 1. The Root Cause: "Warehouse id: 0" In StarRocks 3.x, the Warehouse is a mandatory component for resource management and query execution. * Log Explanation: The

ConnectorHistorgramColumnStatsCacheLoader

is an asynchronous background service that loads statistics for external tables (like the Iceberg tables indicated by your

BaseMetastoreTableOperations

log). * The Conflict: This loader is trying to access the

default_warehouse

(ID 0) to process statistics, but the FE's internal

WarehouseManager

hasn't initialized or registered it yet. This results in the

ErrorReportException: Warehouse id: 0

, which can block the FE from completing its startup sequence and reaching the

Running

status. This is often seen in Kubernetes (K8s) environments when: 1. Leader Election/Initialization lag: The FE pod starting as the Leader hasn't finished creating the default warehouse metadata before background tasks start running. 2. Metadata Inconsistency: If this is a new cluster or a restart after a crash, the metadata image might be missing the warehouse definition, or the

meta_dir

is not being persisted correctly. 3. Shared-Data Mode: If you are using shared-data (compute-storage separation), the FE requires successful communication with the object storage to initialize the default warehouse. 2. Troubleshooting Steps Step 1: Check the Leader FE Check the logs of the first FE pod (usually the one that should be the Leader). If the Leader fails to initialize the

default_warehouse

, all Followers will fail with the same error. * Look for errors like

Failed to initialize default warehouse

or

Master not ready

earlier in the log. Step 2: Temporary Workaround (Disable Stats Loader) To bypass the crash and allow the FE to reach a

Running

status, you can temporarily disable the background statistics collection for external catalogs. Add the following to your

fe.conf

(via your K8s ConfigMap):

properties
# Disable background refresh for connector (external) metadata
enable_background_refresh_connector_metadata = false

# Optional: Disable automatic stats collection entirely to stabilize startup
enable_statistic_collect = false

Note: After the FE starts successfully, you should investigate why the warehouse was missing and re-enable these features. Step 3: Verify Metadata Persistence If your FE pods are "init-looping," ensure that the

meta_dir

is mounted to a Persistent Volume (PV). If the metadata is wiped on every pod restart, the FE will struggle to maintain the cluster state (including the warehouse definitions). Step 4: Check for Known Bugs in 3.5.x Version 3.5.10 is a very recent release (as of the late 2025/2026 timeline). There are known issues where the

default_warehouse

initialization can fail in specific K8s deployment patterns (e.g., Issue #51576). * Ensure that your

fe.conf

does not have conflicting warehouse settings. * If this is a fresh cluster, try clearing the

meta_dir

once and restarting the Leader FE first. 3. Summary of Log Meanings *

BaseMetastoreTableOperations.refreshFromMetadataLocation()

: FE is trying to sync metadata from an external Iceberg/Hive table. *

com.starrocks.common.ErrorReportException: Warehouse id: 0

: The fatal error. The system cannot find the default compute resources needed to perform internal tasks. *

WARN CachedStatisticStorage... Get connector table column statistics filed

: A secondary warning caused by the warehouse missing; the stats engine cannot function without a warehouse. If the problem persists after disabling the stats loader, it indicates a deeper metadata corruption or a network issue between FE pods.

@Rocky addtionaly I can find another WARN logs on fe pod following this. WARN MaterializedView.getDefineQueryParseNode(): 2518] parse original view define sql failed: com.starrocks.sql.analyzer.SemacticException Getting analyzing error. Detail message: Unknown database '~~' at com.starrocks.common.ErrorReport.reportSemanticException and I can find a lot of jdb files and stat.csv file in fe pvc/bdb

The additional logs confirm that your FE is experiencing a metadata initialization failure, likely caused by a dependency loop or a race condition during the startup sequence. 1. Analysis of the New Errors *

Unknown database '~~'

: This occurs when the FE tries to reload and "re-analyze" a Materialized View (MV) during startup. It fails because the database referenced in the MV's original SQL is not yet "visible" to the metadata manager at that specific moment in the boot process. This is common if the MV references an External Catalog (like Hive or Iceberg) that hasn't fully initialized yet. *

stat.csv

and

jdb

files: *

.jdb

files: These are Berkeley DB (BDB) journal files. Having many of them is normal, as they represent your cluster's metadata history. However, if they are not being cleaned up, it may indicate the BDB cleaner is stuck or disk space is low. *

stat.csv

: This is a file used by the Statistics Engine. Its presence in the

bdb

directory suggests the FE is attempting to persist or load table statistics (which matches your previous error about

ConnectorHistorgramColumnStatsCacheLoader

). 2. Solutions to Fix the Init Failure To break the "init loop" and get the FE into a

Running

status, apply the following configurations in your `fe.conf`: A. Ignore Materialized View Errors The FE is currently crashing because it considers the MV parsing failure fatal. You can force it to skip this check:

properties
ignore_materialized_view_error = true

This allows the FE to finish starting even if some MVs have broken references. B. Suppress Background Stats & Warehouse Tasks The

Warehouse id: 0

error happens because background statistics loaders are trying to use the default compute resources before the Warehouse Manager is ready. Disable these temporarily:

properties
# Disable the stats loader that is throwing the Warehouse ID: 0 error
enable_statistic_collect = false
enable_background_refresh_connector_metadata = false

C. Increase BDB Reserved Disk (If applicable) Ensure your FE PVC has plenty of space. StarRocks requires at least 5GB of free space for BDB JE to operate safely. If your disk is nearly full, the BDB cleaner will stop, and

jdb

files will accumulate until the FE crashes.

properties
# Ensure BDB has enough breathing room
bdbje_reserved_disk_size = 536870912 # 512MB (default)

3. Recommended Recovery Procedure 1. Update your K8s ConfigMap for the FE with the parameters in Step 2A and 2B. 2. Restart the FE pods. 3. Once the FE is

Running

, log in via MySQL and check the status of your warehouses and MVs:

sql
    SHOW WAREHOUSES; -- Ensure 'default_warehouse' exists and is 'Ready'
    SHOW MATERIALIZED VIEWS; -- Check for MVs with 'Inactive' or 'Error' status

4. If everything looks stable, you can re-enable

enable_statistic_collect

and

enable_background_refresh_connector_metadata

via

ADMIN SET FRONTEND CONFIG

. Note: If the

Unknown database

error refers to an internal database that should exist but doesn't, you may have metadata corruption. In that case, you might need to recover from a metadata image or re-bootstrap the node using a healthy Leader-Follower replication (the

--helper

method). References • administration/management/FE_configuration.md • administration/Meta_recovery.md • sql-reference/sql-statements/View/SHOW_CREATE_VIEW.md • using_starrocks/async_mv/Materialized_view.md References • administration/Meta_recovery.md

@Rocky FE POd is not running status, its 0/1 running for now. give me another options for handling this problems.

Sorry, I encountered an error while processing your request: 400 INVALID_ARGUMENT. {'error': {'code': 400, 'message': 'The answer candidate length is too long with 62901 tokens, which exceeds the maximum token limit of 8192.', 'status': 'INVALID_ARGUMENT'}}

@Rocky FE POd is not running status, its 0/1 running for now. give me another options for handling this problems.

Since your FE pod is stuck at 0/1 Running, it means the StarRocks FE process is either crashing repeatedly or failing to pass its readiness/liveness probes because it cannot complete its startup sequence. Based on the logs provided, you are hitting a race condition in version 3.5.10 where background metadata tasks (for Materialized Views and Statistics) are starting before the core system (the Warehouse and Catalog) is fully initialized. Here are the additional options to resolve this and get the pod to a

Running

status: Option 1: Forced Bypass via

fe.conf

(Recommended) You can force the FE to ignore the errors that are currently causing it to hang or crash. Update your

fe.conf

(via K8s ConfigMap) with these specific flags: 1. Ignore MV Errors: This prevents the "Unknown database" error from stopping the startup.

properties
    ignore_materialized_view_error = true

2. Disable Stats and Background Metadata: This stops the

Warehouse id: 0

error by preventing background tasks from running during the boot phase.

properties
    enable_statistic_collect = false
    enable_background_refresh_connector_metadata = false

3. Extended Startup Timeout: If you are using the StarRocks Operator, the pod might be killed by K8s before it finishes replaying large metadata logs. Increase the

initialDelaySeconds

for your liveness and readiness probes in the StarRocksCluster YAML. Option 2: Metadata Recovery using the "Helper" Flag If the FE metadata is inconsistent (indicated by the numerous

.jdb

files and the DB lookup failures), you can try to re-sync the metadata from a healthy node (if this is a multi-FE cluster) or force it to re-initialize. * If you have other FE nodes running: In your K8s deployment, you can temporarily modify the start command of the failing pod to include:

./bin/start_fe.sh --helper <leader_fe_ip>:<edit_log_port> --daemon

This will force the pod to pull a fresh metadata image from the leader. * If this is a single FE cluster: You may need to perform a "Metadata Recovery." Back up your

meta_dir

and check the

image/VERSION

file to ensure the

clusterId

matches. If the BDBJE logs (the

.jdb

files) are corrupted, the FE will never reach a

Running

status. Option 3: Handle Disk Space and BDB Logs The "a lot of jdb files" in

fe/meta/bdb

often points to a BDBJE (Berkeley DB) log cleaning failure. 1. Check PVC Space: Run

kubectl exec

(if possible) or check K8s events to see if the disk is full. If the disk is >90% full, BDBJE stops cleaning old journal files, and the FE will fail to start. 2. Clean up stats: The

stat.csv

files are temporary. If they are massive, it may be causing the FE to time out while loading statistics into memory. You can try deleting the

stat.csv

file (after a backup) to see if it allows the FE to bypass the stats-loading phase. Option 4: The "Last Resort" (Metadata Re-bootstrap) If none of the above work and the FE is permanently stuck in

0/1

, you can force the FE to start as a "New" leader: 1. Stop the FE pod. 2. Find the

meta/image/ROLE

file in your PVC. 3. Ensure the role is set to

FOLLOWER

. 4. If the metadata is beyond repair and you have a backup, you can delete the

bdb

directory (NOT the

image

directory) and restart the FE with the

--helper

flag pointing to another node to re-sync. Summary Checklist Action | Purpose

ignore_materialized_view_error = true

| Skip the "Unknown Database" crash

enable_statistic_collect = false

| Skip the "Warehouse id: 0" crash Check PVC Disk Usage | Ensure BDBJE has space to rotate logs Increase Liveness Probe Timeout | Give the FE more than 5-10 minutes to replay logs Note: Once the FE is finally

Running

, you should immediately run

ANALYZE TABLE ...

manually for the problematic MVs and then re-enable the background configurations. References

@Rocky fe.conf which is you give me is not working in this case

ignore_materialized_view_error = true
    enable_statistic_collect = false
    enable_background_refresh_connector_metadata = false

still creating error and WARN message,

If the previous

fe.conf

settings did not resolve the issue and the FE is still stuck at 0/1 Running, it indicates that the FE is failing at a very early stage of metadata replay (likely before those flags even take effect) or that the Berkeley DB (BDB) itself is corrupted. Here are the next steps to recover your FE, ranging from "safest" to "last resort." 1. Metadata Recovery Mode (Recommended) This mode allows the FE to start in a restricted state to bypass many initialization checks. 1. Add this to your

fe.conf

(via ConfigMap):

properties
    metadata_enable_recovery_mode = true

2. Restart the FE. 3. If it starts: You must check the health of your metadata. Run

SHOW PROC '/meta_recovery';

. If it's clean, run

ALTER SYSTEM CREATE IMAGE;

to save a healthy snapshot, then remove the flag from

fe.conf

and restart again. 2. The "Clean Slate" Recovery (Best for Followers) If this FE is a Follower (not the Leader) or if you have a multi-FE cluster: 1. Stop the FE pod. 2. Wipe the metadata directory: Delete the contents of the

meta

folder on the PVC (specifically the

bdb

and

image

directories). 3. Restart with Helper: Start the FE with the

--helper

flag pointing to the IP of a healthy Leader FE. _ _Note: In K8s, this usually involves editing the deployment command temporarily to include

./fe/bin/start_fe.sh --helper <leader-ip>:9010 --daemon

.* 3. Handle the

stat.csv

and

jdb

files The presence of many

jdb

files and a

stat.csv

in the BDB folder often suggests the FE is timing out while trying to load massive statistics or that the BDB cleaner is stuck. 1. Check Disk Space: If the PVC has less than 5GB free, BDBJE will stop cleaning journals and the FE will crash. Increase the PVC size immediately. 2. Remove

stat.csv

: This file is used for persisting table statistics. If it is corrupted, it can cause the FE to hang. You can try moving/renaming this file out of the

meta

directory to see if the FE starts without it. 4. Skip Specific Bad Journals Check the

fe.log

for a line like:

got interrupt exception or inconsistent exception when replay journal <ID>

. If you see a specific ID, you can tell the FE to skip it:

properties
metadata_journal_skip_bad_journal_ids = <ID>

5. Verify Permissions Ensure the user running the StarRocks process has full read/write permissions to the

meta

directory. If the permissions were changed (e.g., during a manual backup), the FE will fail to initialize the BDB environment. Summary Checklist for K8s: * Logs: Run

kubectl logs <fe-pod-name> -n <namespace> --previous

to see why the last container crashed. Look for

InsufficientLogException

or

ChecksumException

. * Probes: Temporarily disable

livenessProbe

and

readinessProbe

in your YAML so K8s doesn't kill the pod while it's attempting a slow metadata recovery. Warning: If you only have one FE node and the metadata is corrupted, always back up the

meta

directory before deleting any files or enabling recovery modes. References • administration/Meta_recovery.md • administration/management/FE_configuration.md • faq/Deploy_faq.md • using_starrocks/async_mv/Materialized_view.md References • administration/Meta_recovery.md • faq/Deploy_faq.md

@Rocky how can i restore fe meta for internal catalog if only left data on s3

In a StarRocks shared-data (compute-storage separated) cluster, the actual data (shards/segments) is stored on S3, but the mapping and schema (the "internal catalog") are stored in the FE's local metadata (BDBJE). If you have lost your local FE metadata and only have the data on S3, your options depend on whether you had a specific backup feature enabled: 1. If "Cluster Snapshot" was enabled (StarRocks 3.4.2+) Starting from version 3.4.2, StarRocks introduced an Automated Cluster Snapshot feature specifically for shared-data clusters. If this was turned on before the failure, a full copy of your FE metadata was periodically saved to your S3 bucket. * How to restore: 1. Deploy a new StarRocks cluster. 2. Configure the

fe.conf

to point to the same

aws_s3_path

and S3 credentials. 3. During startup, if the snapshot exists in that S3 path, the new FE can load the metadata image directly from S3. * Verification: Check your S3 bucket for a folder named

cluster_snapshots

or similar in your StarRocks root path. 2. If you used the

BACKUP

command If you manually executed the

BACKUP

command to an S3 repository before the failure: * How to restore: 1. Set up a new cluster. 2. Create a repository pointing to the same S3 location:

CREATE REPOSITORY ... WITH S3

. 3. Run the

RESTORE

command:

RESTORE TABLE ... FROM ...

. _ _Note: Standard

BACKUP/RESTORE

was not fully supported for shared-data clusters in early 3.x versions; it is primarily a shared-nothing feature or a 3.4+ feature.* 3. If NO Snapshot or Backup exists (Metadata is Lost) If you did not have Cluster Snapshots enabled and you have no local

meta/

directory backup, it is extremely difficult to recover the internal catalog. The data on S3 is stored in a partitioned, ID-based directory structure (e.g., `s3://bucket/path/db_id/table_id/tablet_id/`). Without the FE's metadata: * You lose the mapping of which

table_id

belongs to which table name. * You lose the schema (column names and types). * You lose the data versioning info. Last Resort Steps: 1. Search for an

image.xxxx

file: Check if any FE pod/node still has a copy of the

meta/image/

directory. If you find one, you can use it to boot a new FE. 2. External Catalog (if applicable): If your "internal" data was actually managed by an external metastore (like Glue or Hive) and you were just querying it via StarRocks, you can simply recreate the External Catalog. 3. Manual Reconstruction (Highly Technical): In theory, if you know your schema perfectly, you could try to map the S3 directory IDs back to tables by looking at the file sizes and timestamps, but StarRocks does not provide a tool to "import" existing shared-data shards into a new catalog. Recommendation If this is a production emergency and you are using StarRocks 3.4+, look specifically for the Cluster Snapshot files on S3. If you are on an older version and have no

meta

folder backup, the data on S3 is effectively "orphaned" and cannot be easily re-attached to a new cluster. 0/1 FE. 1 running FE. For future prevention, always ensure

ADMIN SET AUTOMATED CLUSTER SNAPSHOT ON;

is enabled in shared-data mode. References • deployment/shared_data/s3.md