Hi all, I'm trying to optimise a nf-core/mag conf...
# configs
v
Hi all, I'm trying to optimise a nf-core/mag config using slurm+apptainer on our HPC. The basic setup works, but I'm wondering how to specify using the 'local' executor for a specific task: for example, i'm using the following for the host removal run, which submits a job to the cluster as desired:
Copy code
withName: BOWTIE2_HOST_REMOVAL_BUILD {
        cpus   = { 16 * task.attempt }
        memory = { 28.GB * task.attempt }
        time   = 120.h
    }
However, since our cluster accounts per 16 cores, it is inefficient to run single core jobs via the slurm scheduler. The main nf-core/mag process is submitted to a cluster node, also with 16 cores, since running such a process on the login nodes is not permitted. This leaves me with 15 cores unused in that job. I would like to run processes like these 'local' w.r.t. where the nf-core main process runs (is that possible at all), for example processes specified like these:
Copy code
withLabel: process_single {
        cpus   = 1
        memory = { 6.GB * task.attempt }
        time   = 120.h
    }
withLabel: process_low {
        cpus   = { 2 * task.attempt }
        memory = { 12.GB * task.attempt }
        time   = 120.h
    }
n
I think that should be possible by using
executor = local
in the labels. But, there is the major pitfall of the local executor that it actually does not control resource usage, maybe a local executor like
flux
or
hyperQueue
would be more suitable
v
at least in my case something like
Copy code
withLabel: process_low {
        cpus   = { 2 * task.attempt }
        memory = { 12.GB * task.attempt }
        time   = 120.h
        executor = 'local'
    }
is not working. The task still gets submitted to a slurm queue, but maybe this has something todo with the '-resume' option. I'm not sure if changes to the config are effective for resumed jobs
n
they are, but it depends on the priority of the files
v
OK. Thanks. I'm loading my configs using:
Copy code
-profile snellius -c snellius.config -c snellius_mag.config
The first config contains
Copy code
params {
    // Config Params
    config_profile_name        = 'Snellius'
    config_profile_description = 'Dutch National supercomputer hosted at SURF. '
    config_profile_contact     = ''
    config_profile_url         = '<https://servicedesk.surf.nl/wiki/display/WIKI/Snellius>'

    max_time = 5.d
}

process {
    executor       = 'slurm'
    scratch        = false
    
    // Default partitions
    time           = params.max_time
    jobName = {
                n = "${task.name}" -> n.replaceAll('[\\\s:()]', '_')
    }
        
}

/*
 * Apptainer configuration
 * Apptainer is used to run containerised tools.
 */
apptainer {
    enabled     = true
    autoMounts  = true
    // the default is 20 minutes and fails with large images
    pullTimeout = "3 hours"

}

profiles {
    snellius {
        process {
            executor = 'slurm'
            //clusterOptions = '-p genoa'
        }
    
        executor {
                name = 'slurm'
                queueSize = 120
                submitRateLimit = '5 sec'
                jobName = {
                    n = "${task.name}" -> n.replaceAll( '[\\\s:()]', '_')
                }
        }

        singularity.enabled    = false
        singularity.autoMounts = false
        conda.enabled          = false
        docker.enabled         = false
        podman.enabled         = false
        shifter.enabled        = false
        charliecloud.enabled   = false
        apptainer.enabled      = true

    }
}
The second config is an adapted version of the eva.config in short: I've changes all the times to max 5.d (the max of our cluster) I've changed all cpus to chunks of 16 cores / 28 GB or chunks of 16 cores/120GB for the 'fat' nodes https://github.com/nf-core/configs/blob/master/conf/pipeline/mag/eva.config