I had a sarek job running for a few days but it seems to have stopped and has not completed successfully. It might be something related to missing optional files, in the nextflow.log file:

Process NFCORE_SAREK:SAREK:BAM_MARKDUPLICATES:GATK4_MARKDUPLICATES > Skipping output binding because one or more optional files are missing: fileoutparam<1:1>

. Can anyone help here?

I don't think this warning is related to the error

I would add to that that there are similar processes that were killed with exit code

143

and succeeded on retry (by default for some exit codes the processes are retried with more resources). My guess is that -- as Maxime suggests -- a retry with more resources may be successful here as well, but it was not done automatically because there was no exit code to match the ones in the config. Your options are to give the failing process types a custom config with more resources, so they have a chance to succeed immediately, or you modify the

errorStrategy

to

retry

regardless of exit code, which will result in the same failure, followed by a hopefully successful retry with more resources (this is a more general solution that may help with other processes as well). Either way, you would rerun with

-resume

.

Is there a way that I can set resources to max? As Im not sure how much memory to set it to. Is this all done in my

nextflow.config

file? @Thomas Danhorn @Maxime Garcia

An easy way to see what is being allocated and also used is to look at

execution_report_*.html

. It has a process table (use the search box to find the processes of interest) with columns for allocated cpus/memory/time, as well as

peak_rss

(max. memory) and

duration

to see what was really used. (The actual memory metrics are not always given, but you can probably see from the exit code why it failed.) If you have identified which process fails and why, you can make a config file (to be used with

-c

) to set its resources like this:

process {
    withName: 'SOME_PROCESS' {
        time    = 96.h
        memnory = '240 GB'
    }
}

(obviously change the name and values as needed)