@Rocky Explain why it may be beneficial to pause ingestion to my shared nothing zero replication primary key table if i am adding BE nodes. I roughly remember it has to deal with concurrent write and tablet rebalancing

The primary reason it is beneficial to pause ingestion when adding BE nodes to a cluster containing Shared-Nothing, Zero-Replication (replication_num=1) Primary Key tables is that the tablet rebalancing process for Primary Key tables is significantly more resource-intensive and sensitive than for other table models. Here is why concurrent writes during rebalancing can be problematic: 1. Persistent Index and Metadata Synchronization Primary Key tables rely on a Primary Key Index (often persistent on disk) and Delete Vectors to manage updates and deletes. * Migration Complexity: When StarRocks moves a tablet from an old BE to a new BE, it doesn't just copy raw data; it must also transfer or rebuild the Primary Key Index and the associated metadata. * Incremental Catch-up: While the data is being copied, if ingestion continues, the source BE continues to generate new rowsets and index updates. The target BE must then "catch up" by fetching these new incremental versions. If the ingestion rate is high, the migration might take a very long time to finish or fail to catch up entirely. 2. High Resource Contention Primary Key tables are already more resource-intensive during writes because they perform "Delete+Insert" logic (index lookups and delete vector generation). * CPU/Memory Pressure: During rebalancing, the source BE is under heavy I/O and CPU load as it reads data to send to the new node. Concurrent writes compete for the same CPU and memory (specifically the primary key index cache and

transaction_apply

workers). * Compaction Backlog: Migration and high-frequency writes both trigger background tasks. If the BE is overwhelmed, the "compaction score" can skyrocket, leading to ingestion slowdowns or failures (e.g., "too many versions" errors) regardless of the rebalancing. 3. Risk with Zero Replication (

replication_num=1

) In a zero-replication setup, you have no redundancy. * Single Point of Failure: While a tablet is being migrated, the source BE is the only copy. If the source BE becomes unresponsive due to the combined load of migration and heavy ingestion, you risk data unavailability or, in extreme cases, corruption if the node crashes mid-migration. * Switchover Sensitivity: The final stage of tablet migration requires a "handover" where the metadata is updated in the FE. Concurrent writes during this critical window can lead to transaction retries or failures if the tablet state changes during the commit phase. 4. Performance "Stop-and-Go" If you don't pause ingestion, the rebalancing process itself will be much slower. By pausing or reducing ingestion, you allow the cluster to complete the tablet migration quickly. Once the tablets are balanced across the new, larger set of BE nodes, the cluster will have higher total throughput to handle the backlog of data much faster than it could during the migration. Recommendation If you cannot fully pause ingestion, you should at least: * Reduce Ingestion Frequency: Increase the batch size and the interval between loads. * Monitor Migration Progress: Use

SHOW BACKUP

(for clones) or check the FE

sys_log

to see if tablets are failing to migrate due to timeout or version mismatches. * Scale-out during low-traffic windows: Ensure the persistent index has enough disk I/O (SSD) to handle both the read-load of migration and the write-load of ingestion. References • best_practices/primarykey_table.md • table_design/table_types/primary_key_table.md

@Rocky Isn't the main issue that we have no raft consensus with zero replication? If we have replication = 3, concurrent ingestion is fine because 2/3 can agree on the state of the data and it happens much faster.

References • faq/operation_maintenance_faq.md • administration/management/resource_management/Replica.md • best_practices/primarykey_table.md • table_design/table_types/primary_key_table.md

@Rocky Answer my question. Isn't the main issue that we have no raft consensus with zero replication? If we have replication = 3, concurrent ingestion is fine because 2/3 can agree on the state of the data and it happens much faster.

You are correct that the lack of Quorum (Write Quorum) is a central reason why this scenario is so much riskier than a standard

replication_num=3

setup. While StarRocks uses a primary-secondary replication protocol rather than a standard Raft implementation for data, the logic of "majority agreement" (Quorum) is what provides the safety net you are describing. Here is why the lack of consensus/quorum makes the

replication_num=1

case specifically problematic during BE expansion: 1. No "Safety in Numbers" for Transactions In a

replication_num=3

setup, if one BE is under extreme I/O pressure because it is acting as a source for a tablet clone (migration), the transaction can still succeed as long as the other two replicas acknowledge the write. The node being "slowed down" by rebalancing doesn't block the entire ingestion pipeline. With

replication_num=1

, that single BE is the only vote. If the rebalancing load causes that BE to hit a timeout or resource exhaustion, the transaction fails immediately. 2. The "Moving Target" Problem (Version Catch-up) Rebalancing a Primary Key tablet is essentially a "clone" operation. The system copies the data and the Primary Key Index from the source BE to the new BE. * With 3 Replicas: The FE can coordinate the clone from any healthy replica. If ingestion continues, the new replica eventually "catches up" by pulling the incremental versions it missed during the initial copy. * With 1 Replica: Because there is only one source of truth, if ingestion is heavy, the source BE is constantly creating new rowsets and index versions. In a zero-replication environment, if the source BE generates new versions faster than the target BE can pull them over the network, the migration can enter a loop of infinite catch-up or simply time out and fail. 3. Resource Contention on the "Single Point of Truth" Primary Key tables are unique because they require a

transaction_apply

phase where the BE must: 1. Load the Primary Key Index (from memory or SSD). 2. Lookup keys to generate Delete Vectors. 3. Update the index. When

replication_num=1

, the BE is doing all of the above for every ingestion, while simultaneously being scanned by the rebalancing process to send data to the new node. Without a quorum to share the load, this single BE often hits the

tablet_max_versions

limit or

transaction_apply

worker limits, causing the cluster to become unstable. Summary Your intuition is spot on: The lack of a quorum mechanism means there is no "consensus" safety net. * In

replication_num=3

, the system uses the majority to maintain availability and data state, making concurrent ingestion "fine." * In

replication_num=1

, you have a "fragile" single point of failure for that data. Pausing ingestion effectively "freezes" the state of the tablet, allowing the one-and-only copy to be moved to the new node as quickly and safely as possible without competing for the same disk and CPU resources. References • administration/management/resource_management/Replica.md • best_practices/primarykey_table.md • table_design/table_types/primary_key_table.md References • table_design/table_types/primary_key_table.md • best_practices/primarykey_table.md

@Rocky How do I monitor tablet rebalancing when adding BE nodes? Is it SHOW PROC '/statistic'

Yes,

SHOW PROC '/statistic'

is one way to monitor it at a high level, but the more specific and detailed path for rebalancing tasks is under

/cluster_balance

. Here are the primary ways to monitor tablet rebalancing after adding BE nodes: 1. High-Level Status:

SHOW PROC '/cluster_balance'

This gives you a summary of the current balancing state across the cluster. *

pending_tablets

: Tablets waiting to be scheduled for movement. *

running_tablets

: Tablets currently being migrated to new nodes. *

history_tablets

: The total count of completed migrations. 2. Detailed Task Monitoring:

SHOW PROC '/cluster_balance/running_tablets'

This is the most useful command to see exactly what is happening right now. It provides: * TabletId: Which tablet is moving. * Type: Will show

BALANCE

for rebalancing tasks. * SrcBe / DestBe: The source and destination Backend IDs. * Rate: The current data transfer speed. * State: Should be

RUNNING

. 3. Database-Level Summary:

SHOW PROC '/statistic'

As you mentioned, this works well for a quick health check. * Look at the

CloningTabletNum

column. If this number is greater than 0, it means that database has tablets currently being copied or moved. * To see the specific IDs of those tablets, you can drill down using the Database ID:

SHOW PROC '/statistic/<db_id>'

. 4. Checking Cluster-Wide Balance:

SHOW PROC '/cluster_balance/balance_stat'

This command tells you if the cluster considers itself "balanced" yet. * Under the

Balanced

column, it will show

false

if the FE is still planning to move tablets to the new BE nodes. Once it turns

true

, the rebalancing process for that storage medium or distribution type is complete. Pro-Tip: Changing Rebalance Speed If the rebalancing is taking too long (or is causing too much impact on your primary key table ingestion), you can adjust the speed by modifying the FE configuration

tablet_sched_slot_num_per_path

or the BE configuration

clone_worker_count

, though be careful as increasing these will consume more disk I/O and network bandwidth. References • sql-reference/sql-statements/cluster-management/nodes_processes/SHOW_PROC.md • administration/management/resource_management/Replica.md • faq/operation_maintenance_faq.md • administration/management/resource_management/Replica.md • sql-reference/sql-statements/cluster-management/nodes_processes/SHOW_PROC.md • administration/management/resource_management/Replica.md • sql-reference/sql-statements/cluster-management/nodes_processes/SHOW_PROC.md • faq/operation_maintenance_faq.md References • administration/data_migration_tool.md • administration/management/resource_management/Replica.md