<@U0A71G31CDV> StarRocks 3.5.20, deployed in shar...
# questions-and-troubleshooting
a
@Rocky StarRocks 3.5.20, deployed in shared-date mode through kubernetes operator has a problem spinning up with just one replica in shared_data mode producing the following logs: 2026-08-10 164002.308-04:00 INFO (...) [CatalogUtil.loadFileIO():376] Loading custom FileIO implementation: <FILE_IO_IMPLEMENTATION> 2026-08-10 164002.441-04:00 INFO (...) [ConnectorTableMetadataProcessor.registerCachingIcebergCatalog():81] register to caching iceberg catalog on <CATALOG_NAME> in the ConnectorTableMetadataProcessor 2026-08-10 164002.441-04:00 INFO (...) [ConnectorTableMetadataProcessor.registerCachingIcebergCatalog():81] register to caching iceberg catalog on <CATALOG_NAME> in the ConnectorTableMetadataProcessor 2026-08-10 164003.715-04:00 INFO (...) [StarMgrServer.loadImage():247] star mgr image does not exist in <PATH> 2026-08-10 164003.716-04:00 INFO (...) [StateChangeExecutor.runOneCycle():85] begin to transfer FE type from INIT to UNKNOWN 2026-08-10 164003.718-04:00 INFO (...) [StateChangeExecutor.runOneCycle():179] finished to transfer FE type from INIT to UNKNOWN 2026-08-10 164003.773-04:00 WARN (...) [NativeCodeLoader.<clinit>():60] Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 2026-08-10 164003.876-04:00 INFO (...) [HiveMetaStoreClient.openInternal():462] Trying to connect to metastore with URI <METASTORE_URI> 2026-08-10 164003.922-04:00 INFO (...) [HiveMetaStoreClient.openInternal():543] Opened a connection to metastore, current connections: 1 2026-08-10 164003.983-04:00 INFO (...) [HiveMetaStoreClient.openInternal():604] Connected to metastore. 2026-08-10 164004.197-04:00 INFO (...) [HiveMetaStoreClient.openInternal():462] Trying to connect to metastore with URI <METASTORE_URI> 2026-08-10 164004.200-04:00 INFO (...) [HiveMetaStoreClient.openInternal():543] Opened a connection to metastore, current connections: 2 2026-08-10 164004.203-04:00 INFO (...) [HiveMetaStoreClient.openInternal():604] Connected to metastore. 2026-08-10 164004.283-04:00 INFO (...) [BaseMetastoreTableOperations.refreshFromMetadataLocation():190] Refreshing table metadata from new version: s3a://<BUCKET>/<PATH>/metadata/<METADATA_FILE>.metadata.json 2026-08-10 164004.283-04:00 INFO (...) [BaseMetastoreTableOperations.refreshFromMetadataLocation():190] Refreshing table metadata from new version: s3a://<BUCKET>/<PATH>/metadata/<METADATA_FILE>.metadata.json 2026-08-10 164004.295-04:00 INFO (...) [CatalogUtil.loadFileIO():376] Loading custom FileIO implementation: <FILE_IO_IMPLEMENTATION> 2026-08-10 164004.296-04:00 INFO (...) [CatalogUtil.loadFileIO():376] Loading custom FileIO implementation: <FILE_IO_IMPLEMENTATION> 2026-08-10 164005.716-04:00 INFO (...) [GlobalStateMgr.waitForReady():1228] wait globalStateMgr to be ready. FE type: INIT. is ready: false 2026-08-10 164006.852-04:00 INFO (...) [CatalogUtil.loadMetricsReporter():500] Loading custom MetricsReporter implementation: <METRICS_REPORTER_IMPLEMENTATION> 2026-08-10 164006.852-04:00 INFO (...) [CatalogUtil.loadMetricsReporter():500] Loading custom MetricsReporter implementation: <METRICS_REPORTER_IMPLEMENTATION> 2026-08-10 164006.853-04:00 INFO (...) [BaseMetastoreCatalog.loadTable():69] Table loaded by catalog: <CATALOG>.<SCHEMA>.<TABLE> 2026-08-10 164006.853-04:00 INFO (...) [BaseMetastoreCatalog.loadTable():69] Table loaded by catalog: <CATALOG>.<SCHEMA>.<TABLE> 2026-08-10 164007.081-04:00 INFO (...) [InternalData.registerSupportedFormats():70] Unable to register Parquet for metadata files: Cannot find method: register 2026-08-10 164007.633-04:00 INFO (...) [BaseMetastoreTableOperations.refreshFromMetadataLocation():190] Refreshing table metadata from new version: s3a://<BUCKET>/<PATH>/metadata/<METADATA_FILE>.metadata.json 2026-08-10 164007.634-04:00 INFO (...) [BaseMetastoreTableOperations.refreshFromMetadataLocation():190] Refreshing table metadata from new version: s3a://<BUCKET>/<PATH>/metadata/<METADATA_FILE>.metadata.json 2026-08-10 164007.687-04:00 INFO (...) [BaseMetastoreCatalog.loadTable():69] Table loaded by catalog: <CATALOG>.<SCHEMA>.<TABLE> 2026-08-10 164007.699-04:00 INFO (...) [BaseMetastoreCatalog.loadTable():69] Table loaded by catalog: <CATALOG>.<SCHEMA>.<TABLE> 2026-08-10 164007.718-04:00 INFO (...) [GlobalStateMgr.waitForReady():1228] wait globalStateMgr to be ready. FE type: INIT. is ready: false 2026-08-10 164007.807-04:00 INFO (...) [ConnectorTblMetaInfoMgr.addConnectorTableInfo():70] <CATALOG>.<SCHEMA>.<TABLE>:<OBJECT_ID> add persistent connector table info : ConnectorTableInfo {relatedMaterializedViews=[MvId{dbId=<DB_ID>, id=<MV_ID>}]} 2026-08-10 164007.814-04:00 INFO (...) [MaterializedView.setActive():689] set <MATERIALIZED_VIEW_NAME> to active 2026-08-10 164007.822-04:00 INFO (...) [BaseMetastoreTableOperations.refreshFromMetadataLocation():190] Refreshing table metadata from new version: s3a://<BUCKET>/<PATH>/metadata/<METADATA_FILE>.metadata.json 2026-08-10 164007.903-04:00 INFO (...) [BaseMetastoreCatalog.loadTable():69] Table loaded by catalog: <CATALOG>.<SCHEMA>.<TABLE> 2026-08-10 164008.022-04:00 INFO (...) [BaseMetastoreTableOperations.refreshFromMetadataLocation():190] Refreshing table metadata from new version: s3a://<BUCKET>/<PATH>/metadata/<METADATA_FILE>.metadata.json 2026-08-10 164008.032-04:00 INFO (...) [CachingMvPlanContextBuilder.lambda$submitAsyncTask$5():551] async task <TASK_NAME> finished successfully, cost: 6705ms 2026-08-10 164008.075-04:00 INFO (...) [BaseMetastoreCatalog.loadTable():69] Table loaded by catalog: <CATALOG>.<SCHEMA>.<TABLE> 2026-08-10 164008.098-04:00 INFO (...) [BaseMetastoreTableOperations.refreshFromMetadataLocation():190] Refreshing table metadata from new version: s3a://<BUCKET>/<PATH>/metadata/<METADATA_FILE>.metadata.json 2026-08-10 164008.139-04:00 INFO (...) [BaseMetastoreCatalog.loadTable():69] Table loaded by catalog: <CATALOG>.<SCHEMA>.<TABLE> 2026-08-10 164008.205-04:00 INFO (...) [ConnectorTblMetaInfoMgr.addConnectorTableInfo():70] <CATALOG>.<SCHEMA>.<TABLE>:<OBJECT_ID> add persistent connector table info : ConnectorTableInfo {relatedMaterializedViews=[MvId{dbId=<DB_ID>, id=<MV_ID>}]} 2026-08-10 164008.205-04:00 INFO (...) [MaterializedView.setActive():689] set <MATERIALIZED_VIEW_NAME> to active 2026-08-10 164008.213-04:00 INFO (...) [BaseMetastoreTableOperations.refreshFromMetadataLocation():190] Refreshing table metadata from new version: s3a://<BUCKET>/<PATH>/metadata/<METADATA_FILE>.metadata.json 2026-08-10 164008.306-04:00 INFO (...) [BaseMetastoreCatalog.loadTable():69] Table loaded by catalog: <CATALOG>.<SCHEMA>.<TABLE> 2026-08-10 164008.356-04:00 INFO (...) [ConnectorTblMetaInfoMgr.addConnectorTableInfo():70] <CATALOG>.<SCHEMA>.<TABLE>:<OBJECT_ID> add persistent connector table info : ConnectorTableInfo {relatedMaterializedViews=[MvId{dbId=<DB_ID>, id=<MV_ID>}]} 2026-08-10 164008.357-04:00 INFO (...) [ConnectorTblMetaInfoMgr.addConnectorTableInfo():70] <CATALOG>.<SCHEMA>.<TABLE>:<OBJECT_ID> add persistent connector table info : ConnectorTableInfo {relatedMaterializedViews=[MvId{dbId=<DB_ID>, id=<MV_ID>}]} 2026-08-10 164008.357-04:00 INFO (...) [ConnectorTblMetaInfoMgr.addConnectorTableInfo():70] <CATALOG>.<SCHEMA>.<TABLE>:<OBJECT_ID> add persistent connector table info : ConnectorTableInfo {relatedMaterializedViews=[MvId{dbId=<DB_ID>, id=<MV_ID>}]} 2026-08-10 164008.358-04:00 INFO (...) [MaterializedView.checkIsActiveOnLoadBlocking():1431] baseMv: <MATERIALIZED_VIEW_NAME> is reloading, wait it to be reloaded first 2026-08-10 164008.426-04:00 INFO (...) [CachingMvPlanContextBuilder.lambda$submitAsyncTask$5():551] async task <TASK_NAME> finished successfully, cost: 7098ms 2026-08-10 164008.428-04:00 INFO (...) [MaterializedView.checkIsActiveOnLoadBlocking():1431] baseMv: <MATERIALIZED_VIEW_NAME> is reloading, wait it to be reloaded first I have tried ignore_materialized_view_error = true in fe.conf, but to no avail
✅ 1
r
Based on your logs, the StarRocks FE is hanging during the startup sequence because it is blocked while reloading and validating *Materialized Views (MVs)*—specifically those based on external catalogs (Iceberg/Hive). In StarRocks 3.5.x, a new mechanism was introduced to handle MV metadata reloading after loading a metadata image. If you have complex MV chains (MVs on top of other MVs) or external metadata that is slow to respond, the FE can get stuck in a "waiting" state while trying to activate these views, preventing it from transitioning to a
LEADER
state. Recommended Workarounds You can try the following configurations in your
fe.conf
to bypass or mitigate this blocking behavior during startup: 1. Disable Post-Image Reload Cache (Recommended for v3.5.x) A known issue in early 3.5.x versions can cause MVs to reload redundantly or slowly during startup. This is controlled by a new parameter:
Copy code
properties
enable_mv_post_image_reload_cache = false
Note: This parameter was introduced in v3.5.0 specifically to manage this reload logic. 2. Disable Automatic MV Activation Check StarRocks automatically attempts to check and reactivate MVs that might have become inactive due to base table changes. This check can block the FE startup if it triggers extensive external metadata refreshes.
Copy code
properties
enable_mv_automatic_active_check = false
3. Disable Concurrent Prepare (If Deadlocks are Suspected) If the logs suggest a deadlock (multiple threads waiting on the same MV), you can try disabling concurrent preparation:
Copy code
properties
enable_materialized_view_concurrent_prepare = false
Why
ignore_materialized_view_error
failed
The
ignore_materialized_view_error
parameter only tells the FE to continue if an exception/error occurs during MV loading. In your case, the FE isn't hitting an error; it is in a "WAITING" state (
wait it to be reloaded first
), so the "ignore error" logic is never triggered. Suggested Troubleshooting Steps 1. Apply the configs: Add
enable_mv_post_image_reload_cache = false
and
enable_mv_automatic_active_check = false
to your
fe.conf
. 2. Restart the FE: Since you are using the Operator, update the
StarRocksCluster
CRD or the ConfigMap associated with the FE. 3. Check for "Base MVs": The log
baseMv: ... is reloading
suggests you may have nested MVs. If possible, simplify the dependency chain or ensure the external Hive/Iceberg metastore is highly responsive during the cluster's boot window. If the FE still fails to reach a
LEADER
state after these changes, it is recommended to collect a thread dump (
jstack <pid>
) to confirm if there is a specific lock contention in the
GlobalStateMgr
and share it with the community or via a GitHub Issues. References • administration/management/FE_configuration.md