Hi Starrocks community, we’ve been experimenting with Starrocks for quite some time now. An issue hit is for a second time, on two different environments, related to inability to restore a cluster's nodes replica. The version is 3.3.9, Ubuntu 22.04, AWS, shared nothing, 3x frontend nodes and 6x backend nodes, replication 3x with 3x 2nodes in different racks. The backend nodes are organized with rack awareness like so: be*a*1 and be*a*2- labeled rack_*a*, be*b*1 and be*b*2- labeled rack_*b*, be*c*1 and be*c*2- labeled rack_*c*. The tables are also spread according to this rack_a,rack_b,rack_c awareness and it all worked fine, until we had to decommission bea2 and add a new one. The process was: 1.

ALTER SYSTEM DECOMMISSION BACKEND "bea2.reporting.prod.noto.use1:9050";

2. Wait till the node is gone from

"show backends;"

3. Restore the same node, same IP, with empty data directory-

ALTER SYSTEM ADD BACKEND "bea2.reporting.prod.noto.use1:9050"; ALTER SYSTEM MODIFY BACKEND "bea2.reporting.prod.noto.use1:9050" SET ("labels.location"="rack:rack_a");

4. the newly readded node comes alive with new id and state alive. The cluster starts moving around lots of traffic, from bea1 to bea2, the load of the bea2 goes up and it does so for days without ever finishing. The data volume goes up and down all the time.

"show backends"

reports just a few tablet numbers even after days of hundreds of gigabytes being transferred to this node. Could it be a bug or something? Has anyone experienced something like this? Thanks