Hey all, I have a custom config file question. Th...
# configs
z
Hey all, I have a custom config file question. The HPC I am using has several partitions two that I'd like to employ, epyc-64 and largemem. I am trying to incorporate the use for both of these, e.g. in the MAG pipeline to have larger memory jobs like assemblies or bowtie use the largemem partition and smaller jobs to use the epyc-64, I have it set to delineate on time (as epyc-64 has a 2 day limit and largemem has a 7 day limit). Is there a way that I can use the time statement and also write it to submit jobs via memory reuirement? Here is my current config file
Copy code
process {
    executor = 'slurm'
    queue = { task.time <= 2.d ? 'epyc-64' : 'largemem'}
    clusterOptions = '--account=thrash_89 --export=NONE --mail-type=END,FAIL --mail-user=zjhennin@usc.edu'
    maxRetries = 3
    errorStrategy = { task.attempt <=3 ? 'retry' : 'finish' }
    cache = 'lenient'
}
 

singularity {
    enabled = true
    autoMounts = true
}


executor {
    queueSize = 5000
    submitRateLimit = '10 sec'
}

profiles {
  largemem {
    params {
      config_profile_description = 'University of Southern California HPC profile provided by nf-core/configs.'
      config_profile_contact = 'carc-support@usc.edu'
      config_profile_url = 'https://www.carc.usc.edu/'
      max_memory = 1000.GB
      max_cpus = 24
      max_time = 7.d
    }
  }
 
  epyc-64 {
    params {
      config_profile_description = 'University of Southern California HPC profile provided by nf-core/configs.'
      config_profile_contact = 'carc-support@usc.edu'
      config_profile_url = 'https://www.carc.usc.edu/'
      max_memory = 248.GB
      max_cpus = 64
      max_time = 2.d
    }
  }
}
Is this appropriate to use profiles in order to define each partitions max memory, cpus, and time? Also, I see the tutorial allows the choosing of partitions based on CPUs (like he time argument I have in the config file), can it also do so with memory? i.e. something like
Copy code
queue = { task.memory <= 200.GB ? 'epyc-64' : 'largemem'}
p
Profiles would be for specifying manually so if you want it to pick the best automatically, profiles would not be the way. And if you pass this as a config, no need for profiles at all.
z
Thanks! So would it be possible to add a memory argument alongside the time argument? Since there’s several different partitions to be potentially used I’d like the config file to address that based on process memory usage
As each partition has different max memory, cpus, and max time I’m not sure how else to set limits, for example if it sees only the max memory of the large memory node and submits it to the epic 64 because it is less than too days wall time request it may fail from not enough memory
j
You can make the condition as complicated as you want
queue = { time < 7.d && memory >= 512.GB ? 'huge' : time <7.d && memory < 512.GB ? 'big' : time <= 24.h ? 'short' : 'default'   }
For example
Agree profiles are not great for automatic selection
p
"condition as complicated as you want" essentially means that you get to decide. What makes sense differs for an institutional pipeline that would aim to satisfy close to all uses or just something for you - maybe two days are enough for your jobs? Depending on configuration, your slurm cluster may also accept jobs submitted to multiple partitions and pick one that works. Or not. Worth knowing is also that in practice with versions of slurm and nextflow I've seen, it has been possible to override values set with e.g.
process.memory
in
clusterOptions
(e.g. if it's too small or too big). One approach would be to set the values to what works on either partition, if that works then good. Another is to announce the maximum and do something sensible once the requests are known. Or something in between. If it's for running a specific pipeline, you can probably get by using one partition and associated values as default and tuning the processes as needed.
z
thanks!!
Copy code
process {
    executor = 'slurm'
    queue = { time, memory ->
        if (memory > 240.GB && time < 7.d) {
            'largemem'
        } else if (time < 2.d || memory <= 240.GB) {
            'epyc-64'
        } else {
            'largemem'
        }
    }
    clusterOptions = '--account=thrash_89 --export=NONE --mail-type=END,FAIL --mail-user=zjhennin@usc.edu'
    maxRetries = 3
    errorStrategy = { task.attempt <= 3 ? 'retry' : 'finish' }
    cache = 'lenient'
}
So something like this should be sufficient instead of using the profiles
only issue here is the syntax appears to not work
p
They aren't passed so you need to pick up e.g. time and memory from
task
.
j
Sorry they was my fault for the bad example above, Indeed should be e.g
task.memory
z
okay using
Copy code
process {
    executor = 'slurm'
    queue = { task.time, task.memory ->
        if (memory > 240.GB && time < 7.d) {
            'largemem'
        } else if (time < 2.d || memory <= 240.GB) {
            'epyc-64'
        } else {
            'largemem'
        }
    }
    clusterOptions = '--account=thrash_89 --export=NONE --mail-type=END,FAIL --mail-user=zjhennin@usc.edu'
    maxRetries = 3
    errorStrategy = { task.attempt <= 3 ? 'retry' : 'finish' }
    cache = 'lenient'
}
with this adjustment I now get the error that it can't parse the config file
okay my bad I got it running with:
Copy code
queue = { task.time < 7.d && task.memory >= 512.GB ? 'largemem' : task.time <7.d && task.memory < 512.GB ? 'epyc-64' : task.time <= 24.h ? 'epyc-64' : 'largemem'   }
Thank you so much @pontus and @James Fellows Yates for helping a beginner like myself!!
🎉 1