@Allen Li, @Kevin Cai: We are doing the rolling upgrade via operator one BE node at a time. We have couple of tables with replication factor set to 3. While rolling upgrade is happening, we are seeing Broker Load jobs getting cancelled because the replica which went down and after coming back has LVF set. type:LOAD_RUN_FAIL; errortype:SYSTEM_ERROR; msg:Commit failed. txn: 4475 table: emp tablet: 13051 quorum: 0<2 errorReplicas: 13052{be13042 192.168.1.107 st:NORMAL V:682 LFV 1},13053{be:10003 192.168.1.107 st:NORMAL V:680 LFV682},13054{be:10266 192.168.1.107 st:NORMAL V:680 LFV682},13055{be:10269 192.168.1.107 st:NORMAL V:682 LFV:-1}, commitBackends: [10003, 10266] I am seeing a pattern where the replica which went down, when comes back is always lagging that clone task didn't keep up. So what is the recommendation during rolling upgrade with respect to ingestion? BTW, our ingestion rate is 10's of broker loads every minute.

try upgrade one node and check the system tablet health status, wait until all tablets health and upgrade the next one.

@Kevin Cai , Yeah we are doing the same. But since ingestion is active, we always see VERSION_ERROR. So this expected that the replica which is now up after upgrading, will always go into this? During upgrade, does starrocks recommend to stop ingestion?

yes, expected.

Thanks @Kevin Cai. Would slowing down the ingestion help to get the replica out of VERSION_ERROR and then we can move to next BE?