Any ideas why a Nextflow head job is frequently failing to c nf-core #nostupidquestions

Any ideas why a Nextflow head job is frequently fa...

Jennifer Reeve

01/26/2026, 8:59 PM

Any ideas why a Nextflow head job is frequently failing to cancel pending jobs when the pipeline execution fails? More context in 🧵

Jennifer Reeve

01/26/2026, 9:01 PM

Running Nextflow in Slurm with the head job spawning additional Slurm jobs. This is generally working as expected, but at times when the pipeline fails mid-execution the pending/submitted jobs aren't cleaned up. We do have

errorStrategy

set to

terminate

, so I don't think that is the issue. And the same pipeline/config will sometimes clean up submitted jobs and sometimes not. For example, the output from a recent execution reads:

Copy code

ERROR ~ Pipeline failed. Please refer to troubleshooting docs: <https://nf-co.re/docs/usage/troubleshooting>

 -- Check '.nextflow.log' file for details

With the failing process failing nearly 24 hours ago, but the

.nextflow.log

is still updating this morning with:

Copy code

Jan-27 09:58:49.115 [Task monitor] DEBUG n.processor.TaskPollingMonitor - !! executor slurm > tasks to be completed: 4 -- submitted tasks are shown below
~> TaskHandler[jobId: 4390835; id: 55925; name: NFCORE_SAREK:SAREK:BAM_VARIANT_CALLING_GERMLINE_ALL:BAM_JOINT_CALLING_GERMLINE_GATK:GATK4_GENOTYPEGVCFS (joint_variant_calling); status: SUBMITTED; exit: -; error: -; workDir: workdir/01/11afb5dbaa4e3ac5fb9fbdd411a302 started: -; exited: -; ]
~> TaskHandler[jobId: 4390836; id: 55921; name: NFCORE_SAREK:SAREK:BAM_VARIANT_CALLING_GERMLINE_ALL:BAM_JOINT_CALLING_GERMLINE_GATK:GATK4_GENOTYPEGVCFS (joint_variant_calling); status: SUBMITTED; exit: -; error: -; workDir: workdir/0b/2c9a0ddf3e28abc994150df59c9bb8 started: -; exited: -; ]
~> TaskHandler[jobId: 4390837; id: 55922; name: NFCORE_SAREK:SAREK:BAM_VARIANT_CALLING_GERMLINE_ALL:BAM_JOINT_CALLING_GERMLINE_GATK:GATK4_GENOTYPEGVCFS (joint_variant_calling); status: SUBMITTED; exit: -; error: -; workDir: workdir/b6/7f29566782b9910d8541b5303e3d58 started: -; exited: -; ]
~> TaskHandler[jobId: 4390838; id: 55926; name: NFCORE_SAREK:SAREK:BAM_VARIANT_CALLING_GERMLINE_ALL:BAM_JOINT_CALLING_GERMLINE_GATK:GATK4_GENOTYPEGVCFS (joint_variant_calling); status: SUBMITTED; exit: -; error: -; workDir: workdir/c5/6bd517ea8bfb7d2726a1ee648e7ebf started: -; exited: -; ]

Niklas Schandry

01/27/2026, 8:13 AM

I am not sure if this is what is causing your problem, but I vaguely recall there being a specific slurm issue, where in a multi-cluster setup,

scancel

would not be able to cancel on the correct cluster with only

SLURM_CLUSTERS

being set, but required explicit

-M clustername

, which I think nextflow does not do by default. Solutions include

alias scancel=scancel -M clustername

👀 1

Niklas Schandry

01/27/2026, 8:17 AM

https://bugs.schedmd.com/show_bug.cgi?id=9036

pontus

01/27/2026, 8:40 AM

Seems old enough not to be a concern and even if, I don't see nextflow waiting around, hoping that a future cancel would work better. I'm more wondering if the

errorStrategy

takes properly.

Niklas Schandry

01/27/2026, 8:55 AM

Yeah, it is old, but for me this was an issue up until last year, some HPCs do not like updating.. I am not sure if nextflow actually would retry cancelling or if the idea is more along the lines of cancel and then wait until it shows up as cancelled in the queue

pontus

01/27/2026, 9:22 AM

It used to be that you could lag behind a lot (I worked at a site that did) but nowadays... No.

Open in Slack

Previous Next