Hi, I am encountering an issue while processing some samples of approximately 25 GB on the HPC system. The jobs are failing with the message:

terminated for an unknown reason -- Likely it has been terminated by the external system

However: • I do not see any specific error messages in the work directory, including

.command.err

• I have already allocated sufficient resources (CPU, memory, and time), so this does not seem to be an OOM issue.

withName: 'GATK4_MARKDUPLICATES' {
        memory   = { 200.GB * task.attempt }
        time     = { 12.h   * task.attempt }
        cpus     = 16
        disk     = 800.GB
        scratch  = true
        errorStrategy = 'retry'
        maxRetries    = 3

        ext.args = '--MAX_RECORDS_IN_RAM 2000000'
    }

Could you please help me understand what might be causing this? Thank you.

If its the external system like described in your message you have a higher chance of finding the issue by contacting your cluster admin. Reason can include that the cluster specific config is faulty or you send things to the wrong queue or that something happened on your hpc. What happens if you restart the run with

-resume

?

Even if I

-resume

the pipeline, it will again throw the same error.

Can you check the log of the hpc job for one of the terminated processes? For example

90284.hpc01-login01

? How is the cluster configured for running sarek? I suspect that the jobs are not correctly handed to a cluster queue but rather run on one node which will lead to issues but I cannot judge that without knowing your setup.

Resource_List.mem = 262144mb
    Resource_List.ncpus = 16
    Resource_List.ngpus = 0
    Resource_List.nodect = 1
    Resource_List.place = free
    Resource_List.select = 1:ncpus=16:mem=262144mb
    Resource_List.walltime = 48:00:00

And inside the config file the following:

executor {
    name            = 'pbspro'
    queueSize       = 50
    exitReadTimeout = '30 min'
    pollInterval    = '90 sec'
    submitRateLimit = '10/1min'
}

process {
    cpus          = 24
    memory        = 500.GB
    time          = 72.h
    queue         = 'startq'
    errorStrategy = { task.exitStatus in [104, 134, 137, 139, 143] ? 'retry' : 'finish' }
    maxRetries    = 5

    withName: 'GATK4_MARKDUPLICATES' {
        memory   = { 200.GB * task.attempt }
        time     = { 12.h   * task.attempt }
        cpus     = 16
        scratch  = true
        errorStrategy = 'retry'
        maxRetries    = 3

        ext.args = '--MAX_RECORDS_IN_RAM 2000000'
    }
}

In the first look it seems ok for me. I would suggest to ask the admin about this 🙂