<@UGJEZTL4D> What's this for in the PDC profile? `...
# configs
m
@pontus What's this for in the PDC profile?
Copy code
env {
    // Handle java logging on stdout when discovering duplicated cgroups when
    // running in singularity with Lustre mount
    JAVA_TOOL_OPTIONS = "-Xlog:disable"
}
It's causing a tool to crash if I don't disable this. The JVM won't start.
Copy code
Picked up JAVA_TOOL_OPTIONS: -Xlog:disable
  Unrecognized option: -Xlog:disable
  Error: Could not create the Java Virtual Machine.
  Error: A fatal exception has occurred. Program will exit.
p
I don't recall which tool it was, but it would likely be one in rnaseq. Without that, the thing to collect software version would pick up information log lines and output a malformed file causing breakage down the line.
It seems this is no longer needed after the latest upgrade. Also, it seems we can switch to apptainer now and it supports mounting via fuse. I can make a PR.
👍🏽 1
Untested, will try a run of rnaseq with test profile on dardel at least but created the PR already.
Apparently I started sarek instead, so I guess I'll run that also 🙂
And now running for rnaseq.
m
Are you sure this line works?
Copy code
process {
    beforeScript = 'module load PDC apptainer'
}
I'm getting singularity not found if I don't do this before running nextflow
p
Can't check now but maybe missing documentation? The
beforeScript
helps to make sure singularity (or apptainer) is available in jobs. For pulling images, that's done by the main process and there nextflow doesn't offer the same feature (AFAIK). So one needs to load those module before if it's the first time running with those images.
p
I think that it should be possible to get Singularity / Apptainer to pull on the task nodes now instead of the head process: https://nextflow.io/blog/2024/nextflow-2404-highlights.html#oci-auto-pull-mode-for-singularity-and-apptainer
Hmm, I think that there's an error in that blog post 🤔 Should be
ociAutoPull
p
Is that
singularity.ociAutoPull
that is mentioned in the documentation from earlier edge? There it seems to be for OCI images rather than generally.
p
Yup, OCI only
m
How does it work with caching? If there are 5 samtools jobs submitted, what happens? All 5 pull?
p
Apptainer/Singularity does its own caching (but may not be super helpful if run at the same time).
m
> How does it work with caching? Basically,
ociAutoPull
relies on the Singularity/Apptainer engine for the caching of the images. Nextflow does not save any copy. Also, there is no preliminary
singularity pull
executed, download happens as part of of
singularity exec
within each compute task. Note how this reduces duplication of identical images across the filesystem, only the copy in the Singularity/Apptainer cache exists
As regards concurrency of pulls, Singularity/Apptainer takes care of it
👍🏽 1
m
So this means we should also set
SINGULARITY_CACHEDIR
too instead of
NXF_SINGULARITY_CACHEDIR
/
singularity.cacheDir
?
Does that work if multiple people are using that dir as a cache?
m
So this means we should also set
SINGULARITY_CACHEDIR
too instead of
NXF_SINGULARITY_CACHEDIR
/
singularity.cacheDir
?
exactly, with
ociAutoPull
, it is Singularity, not Nextflow that controls caching of the images
Does that work if multiple people are using that dir as a cache?
This is entirely left to Singularity/Apptainer, as long as cache directory permissions are set properly (everyone in the team able to read/write). This is not something that has been tested so far to be honest. Probably a good question for the Sing/Appt folks as well, they might have extra insights
I will see if I can find extra information on this one
👍🏽 1
So, neither Singularity nor Apptainer support caches shared by multiple users. Concurrent pulls by the same users are supported though.
m
Thanks for investigating
p
So, for extra clarity, having a shared
SINGULARITY_CACHEDIR
is not recommended. I also don't see this changing anything, so far I'm still recommending using converted images ("SIF files") rather than oci (e.g.
docker://
).
m
Exactly, no shared cachedir between users. Note how ociAutoPull still creates a SIF under the hood. The main difference is that the pull is now handled by the container runtime in the compute tasks, and as such it reduces duplication of identical images across the filesystem. For instance, the old approach, even with a defined
NXF_SINGULARITY_CACHEDIR
, still creates 2x copies of each image: 1 in the singularity cache (which is there anyway), and 1 managed by Nextflow.
p
Or if it's in a project with 8 users that regularly does
singularity cache clean
, there will be 8 copies instead of 1 🙂
m
very good point! 🙂