Victor de Jager
12/06/2024, 12:51 PMwithName: BOWTIE2_HOST_REMOVAL_BUILD {
cpus = { 16 * task.attempt }
memory = { 28.GB * task.attempt }
time = 120.h
}
However,
since our cluster accounts per 16 cores, it is inefficient to run single core jobs via the slurm scheduler.
The main nf-core/mag process is submitted to a cluster node, also with 16 cores, since running such a process on the login nodes is not permitted. This leaves me with 15 cores unused in that job.
I would like to run processes like these 'local' w.r.t. where the nf-core main process runs (is that possible at all), for example processes specified like these:
withLabel: process_single {
cpus = 1
memory = { 6.GB * task.attempt }
time = 120.h
}
withLabel: process_low {
cpus = { 2 * task.attempt }
memory = { 12.GB * task.attempt }
time = 120.h
}Niklas Schandry
12/06/2024, 1:31 PMexecutor = local in the labels. But, there is the major pitfall of the local executor that it actually does not control resource usage, maybe a local executor like flux or hyperQueue would be more suitableNiklas Schandry
12/06/2024, 1:32 PMVictor de Jager
12/09/2024, 10:34 AMwithLabel: process_low {
cpus = { 2 * task.attempt }
memory = { 12.GB * task.attempt }
time = 120.h
executor = 'local'
}
is not working. The task still gets submitted to a slurm queue, but maybe this has something todo with the '-resume' option. I'm not sure if changes to the config are effective for resumed jobsNiklas Schandry
12/09/2024, 10:36 AMNiklas Schandry
12/09/2024, 10:36 AMVictor de Jager
12/09/2024, 3:20 PM-profile snellius -c snellius.config -c snellius_mag.config
The first config contains
params {
// Config Params
config_profile_name = 'Snellius'
config_profile_description = 'Dutch National supercomputer hosted at SURF. '
config_profile_contact = ''
config_profile_url = '<https://servicedesk.surf.nl/wiki/display/WIKI/Snellius>'
max_time = 5.d
}
process {
executor = 'slurm'
scratch = false
// Default partitions
time = params.max_time
jobName = {
n = "${task.name}" -> n.replaceAll('[\\\s:()]', '_')
}
}
/*
* Apptainer configuration
* Apptainer is used to run containerised tools.
*/
apptainer {
enabled = true
autoMounts = true
// the default is 20 minutes and fails with large images
pullTimeout = "3 hours"
}
profiles {
snellius {
process {
executor = 'slurm'
//clusterOptions = '-p genoa'
}
executor {
name = 'slurm'
queueSize = 120
submitRateLimit = '5 sec'
jobName = {
n = "${task.name}" -> n.replaceAll( '[\\\s:()]', '_')
}
}
singularity.enabled = false
singularity.autoMounts = false
conda.enabled = false
docker.enabled = false
podman.enabled = false
shifter.enabled = false
charliecloud.enabled = false
apptainer.enabled = true
}
}
The second config is an adapted version of the eva.config
in short:
I've changes all the times to max 5.d (the max of our cluster)
I've changed all cpus to chunks of 16 cores / 28 GB
or chunks of 16 cores/120GB for the 'fat' nodes
https://github.com/nf-core/configs/blob/master/conf/pipeline/mag/eva.config