http://starrocks.io logo
Join Slack
Powered by
<@U0A71G31CDV> Hi, when running `SHOW PROC '/dbs/<...
# questions-and-troubleshooting
j
@Rocky Hi, when running 
SHOW PROC '/dbs/<db>/<tbl>/partitions';
in starrocks I get that Buckets = 5 and ReplicationNum = 3 for each partition. Most of the partitions have 5x3 = 15 entries but there are some partitions for example 
SHOW PROC '/dbs/<db>/<tbl>/partitions/<part>/<idx>';
that only have 5 entries (1 entry per tablet), indicating that there is only a single replica for each tablet. How can it only have a single replica if it was configured to have 3 from the query before? Is this a bug in StarRocks?
r
This behavior typically indicates that some of your partitions are under-replicated. In StarRocks, when you have 
Buckets = 5
and 
ReplicationNum = 3
, the system is expected to maintain 5 tablets, each with 3 physical replicas, for a total of 15 replicas across your Backends (BEs). If 
SHOW PROC '/dbs/<db>/<tbl>/partitions/<part>/<idx>';
shows 15 rows, it is listing all the replicas for those 5 tablets. If it only shows 5 rows for certain partitions, it means those tablets currently have only one replica each. This is usually not a bug, but a state of the cluster. Here are the most common reasons why this happens: 1. Insufficient or Offline Backends StarRocks cannot create 3 replicas if there aren't enough healthy BE nodes available. • Check node status: Run 
SHOW BACKENDS;
and verify that at least 3 BEs are 
Alive
and not in 
Decommission
state. • If you recently lost BEs or if some BEs are unreachable, StarRocks will be unable to satisfy the 
ReplicationNum = 3
requirement for new partitions or repair old ones. 2. Replication in Progress (Pending Repair) If the partitions were recently created (e.g., via dynamic partitioning or a new load), the FE (Frontend) might still be in the process of scheduling the creation of the additional replicas. • Check Tablet Health: Run 
SHOW PROC '/statistic';
to see the 
UnhealthyTabletNum
. If this number is greater than zero, the cluster is aware that some tablets are under-replicated and is likely trying to fix them. • Check Tablet Details: Pick a 
TabletId
from the 5 entries you see and run:
Copy code
sql
  SHOW TABLET <tablet_id>;
This will show you exactly how many replicas exist and which BEs they are on. 3. Creation-time Backend Availability When a partition is first created (especially during a load or a DDL), StarRocks tries to create all replicas immediately. If only 1 BE was available at that exact millisecond, it might have created only 1 replica. The Tablet Scheduler should eventually notice this and clone the missing replicas once more BEs are available. • You can check the scheduler's status with:
Copy code
sql
  SHOW PROC '/cluster_balance';
4. Shared-Data (Cloud Native) Tables If your cluster is a Shared-Data (Cloud Native) cluster, the data is stored in object storage (like S3). In this mode: • Tablets do not have multiple physical replicas on BE disks because the object storage provides the redundancy. • However, if you see 15 entries for some partitions in the same table, this table is likely a traditional Shared-Nothing table, so this point probably doesn't apply. Recommended Next Steps: 1. Check for unhealthy tablets:
Copy code
sql
    ADMIN SHOW REPLICA STATUS FROM <tbl> PARTITION (<part>) WHERE status != 'OK';
2. Verify Backend count: Ensure 
SHOW BACKENDS;
shows enough 
Alive
nodes. 3. Check FE Logs: Look for "tablet" or "clone" related messages in 
fe.log
to see if there are errors preventing the creation of new replicas (e.g., "no available backend" or "disk capacity limit reached"). If the BEs are healthy and the number of entries doesn't increase over time (10-20 minutes), you may need to manually trigger a repair or check if there are specific disk space issues on the remaining BEs. References • sql-reference/sql-statements/cluster-management/nodes_processes/SHOW_PROC.md • sql-reference/sql-statements/table_bucket_part_index/SHOW_PARTITIONS.md • sql-reference/information_schema/partitions.md
In StarRocks, finding a partition with fewer replicas than its configured 
replication_num
is typically not a bug, but rather an indication that the partition is in an under-replicated state. If 
replication_num
is 3 and you only see 5 entries (1 per tablet) instead of 15, StarRocks is aware it should have 3 replicas but currently only has 1. Here is why this happens and how to troubleshoot it: 1. Insufficient or Offline Backends (BEs) To maintain 3 replicas, you must have at least 3 healthy and active Backends. * Check: Run 
SHOW BACKENDS;
. * Reason: If you only have 1 or 2 BEs 
Alive
, or if BEs are in the 
Decommission
state, StarRocks cannot create the 3rd replica. It will create as many as possible (in your case, 1) and wait for more nodes to become available to fulfill the requirement. 2. Disk Space or Resource Exhaustion If a Backend's disk is nearly full (usually >90% or the 
capacity_used_percent_high_watermark
threshold), StarRocks will stop scheduling new replicas to that node. * Check: Look at the 
AvailCap
and 
UsedPct
columns in 
SHOW BACKENDS;
. * Reason: If multiple BEs are full, the system cannot find a suitable home for the missing replicas. 3. Creation-time Availability If the partition was created during a period where only one BE was healthy (e.g., during a cluster rolling restart or a network partition), StarRocks might have successfully created the tablet on the only available node to allow the data load to proceed. * Check: Run 
SHOW PROC '/cluster_balance/pending_tablets';
. * Reason: The FE (Frontend) has a background process called the Tablet Scheduler that periodically checks for under-replicated tablets. If it finds any, it adds them to a queue to "repair" them by cloning replicas. If the queue is backed up or the scheduler is disabled, the replicas will remain at 1. 4. Tablet Health and Repair Status You can verify exactly why StarRocks thinks these tablets are "unhealthy" by running:
Copy code
sql
-- Replace <db> and <tbl> with your actual names
ADMIN SHOW REPLICA STATUS FROM <db>.<tbl> WHERE status != 'OK';
If the status is 
MISSING
, it confirms the replicas are absent. You can then check the global statistics:
Copy code
sql
SHOW PROC '/statistic';
Look for 
UnhealthyTabletNum
. If this is greater than 0, the cluster is currently working on (or failing to) repair these tablets. 5. Shared-Data (Cloud Native) Clusters If you are using StarRocks in Shared-Data mode (storing data on S3/OSS), the behavior is different: * Physical replicas are not stored on BE disks; the storage layer handles redundancy. * However, if other partitions in the same table show 15 entries, you are likely using a Shared-Nothing table, and the "Under-replicated" explanation above applies. Summary of what to do: 1. Verify you have at least 3 BEs in 
Alive
state. 2. Check for full disks on your BEs. 3. Check 
SHOW PROC '/cluster_balance/pending_tablets';
to see if repair tasks are failing. Look at the 
ErrMsg
column there for specific errors like "no available backend" or "low priority high-lighting the root cause. It is rare for this to be a bug; it is almost always a result of the FE's Tablet Scheduler being unable to find a valid Backend to host the additional copies of the data. References • administration/management/resource_management/Replica.md
j
We are using shared nothing architecture and these are async materialized views. All the BEs are healthy at the moment. The partition that has issues is from many months in the past. When the materialized view refreshes every 15min, would it not correct the missing replicas from before?
Open in Slack
PreviousNext
Powered by