Hey folks, Did you face an issue with routine loads become PAUSED due to pods being rescheduled (karpenter) or restarted? I already have pdb for FE and CN pods, so I'm thinking about increasing

loop_count_wait_fragments_finish

to wait more when CN process exits and try to update operator version to latest one (current is 1.10).

you may give a try, this configuration is mostly worked for queries, but not for data loading.

could it also be related to unregistered CN pods? It looks like i have similar issue, when CN pods are scaled down, they are just staying there as DISCONNECTED. It is really hard to catch the reason for PAUSED jobs, but logs clearly says paused due to BE/CN node is not alive. Sometimes jobs are PAUSED after reschedule or restarted, sometimes works ok

yeah, could be related, the retry logic need to be checked again.

our cluster run on spot instance, some routine load will pause when pod is recreate. I create a script resume all when a routine load pause. This is trigger when a keda scale object observe pause routine load metrics > 0