Setting Up nf-core on SLURM HPC: Hi, I am a new ...
# configs
a
Setting Up nf-core on SLURM HPC: Hi, I am a new nf-core user and will be using nf-core on our HPC that uses SLURM. For now, it's just me, but other users from our center will join in the future, so I need to set up everything necessary. I have already run some pipeline tests using a Singularity profile, and they ran successfully. However, I have some questions regarding extending usage to other users: 1. Institutional Configuration Profile: - Is it necessary to create an institutional configuration profile? I am wondering if it is easier (and if it will work) to run the pipelines using the Singularity profile and setting the parameters in a script to submit them as an sbatch job. For example, if I use a Singularity profile without providing any personal configuration file and submit it as an sbatch job with the script specifying the execution parameters for the cluster (e.g., memory, CPUs, max time), which configuration will take precedence? The Singularity default configuration or the script with the SLURM parameters? 2. Configuration File Needs: - If an institutional configuration profile is needed but we do not have it yet, and SLURM scripts are not a solution, do we need our own configuration file to use when running the pipelines? I think setting the parameters in the SLURM script is a good idea because it allows users to select the parameters each time depending on their needs instead of changing or having multiple configuration files. 3. Nextflow and Singularity cache: - Is it necessary to set up a path for all the Singularity and Nextflow cache? I can do it in my home directory, but for multiple users, it might need to be set up in another location on the HPC. What would be the best practice for this? Thank you in advance for your help!
p
1. Not necessary, no. It is a convenience for users. Typically
-profile mycluster
is easier than writing an sbatch job each time. Configuration precedence in Nextflow is here, but just to confirm - if you launch Nextflow in an
sbatch
job then the resources you give it won't matter. Nextflow itself will be submitting more
sbatch
jobs and they won't be affected by the resources assigned to the head Nextflow job. Also note that many people run Nextflow interactively as the head job shouldn't take much in the way of resources (though approach to this varies sysadmin to sysadmin and HPC to HPC). 2. Institutional configs typically never contain parameters for specific pipeline runs, only things related to the cluster. So yes, users should set the parameters each time. Typically we suggest that people do this in a YAML file in the launch directory and do
nextflow run xxx -profile mycluster -params-file myparams.yml
where the profile is the cluster specific config from the institutional configs and the params file are specific to this run. 3. Not necessary, defaults will be used. But on HPC it can be nice to set up a shared cache to prevent many users downloading the same (singularity) images again and again. You can do this using
$NXF_SINGULARITY_LIBRARY
which is allowed to be read-only, so you can put images there for end users.
$NXF_SINGULARITY_CACHEDIR
should then be writeable and Nextflow will fetch and save images there as needed. This is different to
$SINGULARITY_CACHEDIR
which is used by Singularity itself when building images. That also needs to be writeable.
👍 1
Hope that helps!
I'd recommend taking a look at some other institutional configs to get a feel for the kind of things people put in them: https://nf-co.re/configs
Many also include usage documentation, which also gives an insight into how other people set up their shared HPC systems for users
j
a
Thank to you both!!
👍 1
👍🏻 1
s
Configuration precedence in Nextflow is here, but just to confirm - if you launch Nextflow in an
sbatch
job then the resources you give it won't matter. Nextflow itself will be submitting more
sbatch
jobs and they won't be affected by the resources assigned to the head Nextflow job.
@Phil Ewels, is this correct, without explicitly having specified slurm in a config file? I would have assumed that Nextflow would just act locally on the node that it lands on in this case.
p
Exactly - I was making the assumption that the executor was set to
slurm
already. You're right that it defaults to
local
👍🏻
s
Ok. Just clarifying as the original message was without specifying any additional config
👍🏻 1
gratitude thank you 1