Another config question, hopefully the last one be...
# configs
p
Another config question, hopefully the last one before I submit our institutional config! We have a partition with two different node types, normal and high memory, which can be selected using the
sbatch --constraint=mem256/mem128
option. I've added these to the profile's process directive as follows:
clusterOptions = { task.mem >= 112.GB ? "--constraint=mem256" : "--constraint=mem128"}
(112 because we need to account for ~16 GB buffer on our HPC). My question is whether I should do the same for the
max_memory
directive, or if I should just set it to the higher value (i.e. 240 GB). From reading the docs I've understood that this value is used to restrict the resource requests of specific tasks, but I'm not sure if
task.mem
is based on the value of the task/process, or if it set after comparing the requested value with the max_memory threshold?
p
max_memory
should be 240, otherwise nextflow (the nf-core configurations) would limit all tasks to 112. (Actually, I'd assume
task.mem
is not available when
max_memory
is calculated as well, but still.)
p
Gotcha, thanks!
p
I think that you need this:
Copy code
clusterOptions = { check_max(task.mem) >= 112.GB ? "--constraint=mem256" : "--constraint=mem128"}
Where
check_max
is this function (depending on where you're setting this config, you may find that you need to copy + paste this function into that config file, if it doesn't work)
I don't think you need to do anything with
params.max_memory
Ah right, except bumping it as the default is too low as Pontus said, sorry
But without
check_max
the
max_memory
params doesn't do anything anyway 😅
p
I think
task.mem
will already be limited because
process.memory
uses
check_max
in nf-core.
p
ahh because this is
process.clusterOptions
and not
process.memory
I'd be a bit paranoid about order of config resolution here 😬
But ok, if it works 👍🏻
I stand corrected 🙇🏻
FYI there's a PR in progress to move the capping to core Nextflow functionality. It's been languishing a bit, need to bump it up.
👍 1
p
I just found out that the
constraint
option does not seem to work on our HPC 😞 Will reach out to the support team to let them know. Apparently I can access the high memory nodes via a different partition name though, so I ended up just adding additional profiles for high mem nodes. Thanks for the info on the check_max function though!
p
Yes, how that works will depend. Doesn't hurt to check with your systems people, but it might be useful to know you can (slurm allows you to) specify multiple partitions at the same time (comma separated list). [One can setup the system do croak on that, but might be worth trying.]
👍 1