Hello folks, I'm working on a huge pipeline on a supercomputer, it's so big that it triggered the slurm RPC count limit, and somehow over-crowded the slurm database with its huge amount of tasks (as slurm jobs). I tried limiting it's submission rate and poll intervals, but those don't really help with this issue. If I limit it too much, it would be too slow and leaving lots of nodes on the HPC cluster idle. It's pretty difficult to find a trade-off, I wonder if anyone here have had some similar experience?

Did you try modifying the queue size? That's the other control lever that you can typically play with, alongside submit rate etc.

Hi Phil, thanks for the reply! Yes I also tried the queueSize, it still has the same problem. I set the queue size to the max job count of all the available node pool, but due to slurm RPC limit, the actual running & pending jobs are just 1/5 of the queueSize.

If your issue is that you are submitting lots of the same jobs, maybe the array directive can help