Hi all, My run failed because of a node fail in one of the jobs. Specifically NFCORE_SAREKsarekFASTQ_ALIGN_BWAMEM_MEM2_DRAGMAP_SENTIEON:BWAMEM1_MEM After resubmitting the whole run with -resume, sarek finds all of the jobs to be cached so it skipped to the mark duplicates step. I stopped it manually, deleted the necessary work folder for the job that failed and submitted again. Now it can see that it was not done and runs it again. Is this how it should be?

No, Nextflow usually checks both the exit code of the task and if all the specified output files are present in the task working directory. For me this looks like a failure on the cluster side after the task has completed already, i.e. in the job clean up phase of the scheduler. It shouldn't have negatively affected your resumed run in any way.

Hey thanks for the response. Hmm..could be I will look more into it. It seems like this job ended in less time than the rest for the other samples. What I did it should be correct, right?

Deleting the task work directory to force a rerun of that task is the safe option. Would be interesting to see what the status of that first attempt was. You could use

nextflow log -f hash,name,exit,status <run_id>

to get some information about the task status.

Hello, status was FAILED after the node died. From slurm State : CANCELLED,NODE_FAIL

And what task status do you get from nextflow log? Is the task registered there as failed?

Yes on execution trace is registered as failed.