Hi all, I'm writing an institutional config file f...
# configs
p
Hi all, I'm writing an institutional config file for our HPC, but I'm a bit puzzled by the behaviour of the
env
directive and the `APPTAINER_*`/`SINGULARITY_*` environment variables for setting the cache and tmp dir (not the
NXF_*
one housing the final images, but the apptainer ones). I've seen a number of example configs (and threads on slack) that use the
env
directive to set these, e.g.
Copy code
env {
    APPTAINER_TMPDIR="$HOME/.apptainer/tmp"
    APPTAINER_CACHEDIR="$HOME/.apptainer/cache"
}
However, in my own testing, these environment variables are not respected. Manually exporting the variables to my shell before launching nextflow does work however. Digging through the docs, I found [the following](https://www.nextflow.io/docs/latest/config.html#scope-env):
The
env
scope provides environment variables to tasks, not Nextflow itself. Nextflow environment variables such as
NXF_VER
should be set in the environment in which Nextflow is launched.
So reading that, am I correct in assuming that the pulling of images and/or converting docker to apptainer, happens in the main nextflow process, prior to any tasks, and thus cannot access any variables defined by
env
?
p
I believe that this is the case, yes šŸ‘šŸ» At least off the top of my head I would be surprised if setting those variables in a Nextflow config had any effect.. šŸ¤”
p
Thanks for the confirmation! I'll go ahead and describe the behaviour in my config's docs too then, in case others are confused too in the future :)
p
Yes, as currently implemented, pulling of container images are not in the tasks themselves but in the main nextflow monitoring process. Also, should one argue that
APPTAINER_TMPDIR
is set to set
TMPDIR
within the container, the correct way of doing that is to set
TMPDIR
-
env
scope environments are evaluated within the container context. So things like that are just plain wrong.
e
Could probably remove the
env
p
cacheDir
actually sets
NXF_SINGULARITY_CACHEDIR
which indeed does work for me too. But this is only used by nextflow to store the final pulled/built images, not as a cache during pulling/building (which is what
SINGULARITY_CACHEDIR
and
SINGULARITY_TPMDIR
do).
sad cowboy 1
p
That is also a common confusion...
e
p
> That is also a common confusion... It took me a while to grok what was going on too, if I'm being honest šŸ˜‰ I was really caught off-guard by the fact that the TMPDIR is cleaned automatically. I couldn't figure out why it wasn't being used, until I used
watch
and saw what was happening.
As did the fact that the apptainer engine pulls in docker images and converts them, whereas the singularity one pulls in pre-built images from galaxy.
But with this out of the way, I'm slowly getting there.
p
That's only due to a bug/lack of updates in templates.
šŸ‘ 1
So not a difference in the nextflow code, but in nf-core. But so far I don't know of any good reason one would need to use the apptainer support rather than the singularity.
e
Wait. So you're saying if you're using apptainer, it's pulling the docker images?
Going to need to dig into how we're doing the CI then, I think we're using apptainer but then
singularity.enable
...
So, if using
apptainer.enabled
, it will yield what nextflow takes as a docker URL, which will be pulled as such.
šŸ‘šŸ» 1
p
It's documented here @Edmund: https://nf-co.re/docs/usage/installation#pipeline-software (under apptainer).
So in case you are using the singularity profile (
singularity.enable
), but have apptainer installed on your system (aliased to singularity), pre-built singularity images will be used. It is only when you use the
apptainer.enable
type profile that docker images will be converted.
e
Gotcha. So just to make sure I'm understanding all of our module tests are valid and the singularity tests are testing the singularity images?
p
Yes.
p
Follow-up question on this topic: I stumbled across https://github.com/nextflow-io/nextflow/issues/2685 , where they mention making use of
envWhitelist = ['SINGULARITY_TMPDIR']
to allow nextflow to access bashrc variables. However, that doesn't seem to be necessary in my tests. What gives?
p
Again, a better way of doing that is just putting your
TMPDIR
in
env
.
p
Sorry, I'm lost now šŸ˜… Let me recap: •
SINGULARITY_TMPDIR
defines a directory that is used during the build / conversion of singularity images. • It is populated during the master nextflow process, outside of any process tasks. • The
env
scope only applies to process tasks. • So to make nextflow use it, I though it had to be defined as a global environment variable on whatever system you are running the pipeline. What's the use case of defining
TMPDIR
in
env
then according to you?
p
Sorry, I misthought/misread.