Hey @Rocky, yesterday I changed the way the BE pods register to the FE pods: I removed the namespace component of the URL so that they can be restored in any other namespace. My question: when doing that, my lone BE found itself registered TWICE in the FE: once with the older URL (which works) and once with the new URL (which works too). The issue was that the FE was seeing 2 backends and cannot seem to distinguish that they are the same. Now the FE issues errors like this: Failed to get scan range, no queryable replica found in tablet=10156 replica=100023253/ 1/3253/1878BAD:ALIVE, schema_hash=463297183 version=3253 And a SHOW TABLETS FROM mytable; lists tablets but with size 0bytes. I just saw in my metrics that at the exact moment the BE started with its new hostname, at registration it removed all the existing data in its data/ folder. So I lost data. I'm using the Starrocks Operator to manage my cluster, so I don't have much control on how the pods are scheduled. Do you have any advice on limiting the data loss incurred when changing hostnames?

Hello! I'm Rocky. I see you've run into a situation where a change in backend (BE) registration led to duplicate entries in the frontend (FE) and, unfortunately, data loss. Based on how StarRocks handles node identities and how the StarRocks Operator manages pod lifecycles, here is a breakdown of what likely happened and how to prevent this in the future. Why the BE Registered Twice StarRocks identifies a Backend by its host:heartbeat_port string. * Old Identity:

be-pod-0.be-service.namespace.svc.cluster.local:9050

* New Identity:

be-pod-0.be-service:9050

(or similar) Even if it is the exact same physical Pod, the FE sees these as two distinct nodes. Because the Operator or your manual configuration likely "added" the new hostname, the FE assigned it a brand new

BackendId

. The FE now thinks you have two backends: one that is "Missing" (the old hostname) and one that is "New/Empty" (the new hostname). Why the Data was Wiped There are two main reasons why your

data/

folder was cleared: 1. Missing Persistent Volumes (Most Likely): By default, if

storageVolumes

are not explicitly configured in the StarRocks Operator CRD, the operator uses

emptyDir

for storage. Since changing the registration URL required updating the Pod's environment or configuration, the Operator performed a Rolling Update, which restarts the Pod. In Kubernetes, an

emptyDir

is completely wiped whenever a Pod is restarted or rescheduled. 2. Identity/Cluster ID Mismatch: StarRocks BEs store a

cluster_id

and metadata in their storage path (usually in the

meta

subfolder). When a BE starts up and heartbeats to the FE, it receives cluster information. If the BE registers as a new node (new ID) but finds metadata on the disk belonging to an old node ID or a different cluster identity, it may treat the existing data as invalid or "stale" and re-initialize the storage directory to ensure consistency with its new identity. How to Limit Data Loss and Prevent This To safely change hostnames or migrate namespaces in the future, follow these best practices: 1. Always use Persistent Volumes (PVs) Ensure your

StarRocksCluster

YAML has

storageVolumes

defined for BEs. This ensures that even if the Pod restarts or the hostname changes, the physical data persists on the disk.

yaml
starRocksBeSpec:
  storageVolumes:
  - name: be-data
    storageClassName: your-storage-class
    storageSize: 100Gi
    mountPath: /opt/starrocks/be/storage

2. Use

ALTER SYSTEM MODIFY BACKEND

If you must change the hostname of an existing BE, do not simply add the new one. Use the following SQL command to tell the FE that the "identity" of the existing Backend has moved to a new host. This preserves the

BackendId

and maps the existing tablets to the new URL:

sql
-- Syntax: ALTER SYSTEM MODIFY BACKEND HOST "old_host" TO "new_host";
ALTER SYSTEM MODIFY BACKEND HOST "be-0.be-service.old-ns.svc:9050" TO "be-0.be-service:9050";

Note: This is the safest way to "rename" a node without the FE treating it as a brand-new, empty node. 3. Enable FQDN Mode If you are moving across namespaces, it is highly recommended to enable FQDN access. When starting your FE for the first time, you should use the

--host_type FQDN

flag. For existing clusters, you can transition to FQDN mode following the StarRocks FQDN documentation. 4. Increase Replica Count Running a "lone BE" (1 replica) is extremely risky. In StarRocks, if you have

replication_num = 1

and that single BE loses its data or identity, there is no way for the cluster to "self-heal" by cloning data from elsewhere. If you had 3 replicas, the FE would have seen the "new" BE as empty and automatically started cloning data from the other 2 healthy replicas. Summary of Next Steps

Ok @Rocky I have persistent storage, of course. And the clusterID didn't change, only the BackendID. But I understand the issue: I should have renamed the node before the update so that the new registration name was already there.

Actually, you've hit on a very specific behavior of how StarRocks manages node identity. Even with persistent storage, changing a hostname without the proper SQL "rename" command causes a chain reaction that leads to the FE treating your BE as a complete stranger. Why the Data was "Wiped" In StarRocks, a Backend is identified by its host:port string, but internally it is tracked by a unique BackendID. 1. Identity Mismatch: When you changed the URL, the FE registered a new BackendID for that "new" hostname. 2. Metadata Conflict: Your persistent disk still contained metadata (in the

meta/

and

data/

directories) associated with the old BackendID. 3. The "Re-initialization": When the BE process starts up, it heartbeats to the FE. The FE says: "You are Backend ID 10005." The BE looks at its disk and sees: "But my disk says I am Backend ID 10002." * To prevent data corruption or "phantom" data from appearing in the wrong place, the BE will often fail to start or, depending on the version/configuration, it may re-initialize the storage directory if it believes it is joining as a brand-new node to an existing cluster. 4. The Lone BE Trap: Because you have only one BE (

replication_num = 1

), there is no other copy of the data. The FE now sees two backends: *

ID 10002

(Old hostname): Dead, holds all the tablet metadata. *

ID 10005

(New hostname): Alive, holds zero tablets. * Result: All queries fail because the FE is looking for the data on a node that doesn't exist anymore. How to Prevent This in the Future The StarRocks Operator manages the Pods, but the logical registration of the node identity must be handled via SQL to ensure continuity. 1. The Correct Way to Rename (The "Handover") If you need to change the hostname/URL of your BEs, you must perform an

ALTER

command before or during the update so the FE knows the old identity is moving to a new address. This preserves the

BackendID

and the tablet mapping. The SQL Command:

sql
-- Syntax: ALTER SYSTEM MODIFY BACKEND HOST "old_host" TO "new_host";
ALTER SYSTEM MODIFY BACKEND HOST "be-0.starrocks-be.old-ns.svc.cluster.local" TO "be-0.starrocks-be";

Note: Run this for each BE before the Operator restarts the Pods with the new configuration. 2. Use FQDN Mode To avoid having to change hostnames when moving namespaces or clusters, you should enable FQDN access mode. * In the Operator CRD, you can set the

host_type

or ensure the FE is started with the

--host_type FQDN

flag. * This makes StarRocks rely on the DNS names rather than just IP addresses, but you still need to use the

MODIFY BACKEND

command if the actual string of that FQDN changes (e.g., removing the namespace part). 3. Redundancy is Key (The "3-Node Rule") Running a single BE node is highly discouraged for production or any data you care about. * If you have 3 BEs and

replication_num = 3

, you could have deleted the old node, added the new one, and the FE would have simply re-replicated the data from the other two survivors to the new one automatically. * With 1 BE, you have no safety net. Can you recover the data? If the BE actually cleared the

data/

directory on the persistent volume (which it often does when it senses a brand-new identity initialization), the data is unfortunately gone unless you have a snapshot of the volume. Checklist for Recovery: 1. Check the PV: Exec into the pod and check

/opt/starrocks/be/storage

. If it is truly empty, the BE re-initialized it. 2. Clean up FE state: Run

SHOW BACKENDS;

. You will likely see both the old and new URLs. You should

ALTER SYSTEM DROP BACKEND "old_host:port";

to remove the dead reference. 3. Restore: If you have a StarRocks backup (via Broker Load/Backup tool) or a cloud disk snapshot, now is the time to use it.