Hi! Had an issue a couple of months ago when running nf-core/ampliseq on PDC's Dardel. I ran ampliseq v2.17.0 in screen as follows (Nextflow v25.10.4):

nextflow run nf-core/ampliseq \
  -r 2.17.0 \
  -params-file nf-params.json \
  -profile pdc_kth \
  -work-dir $SCRATCH/nxf_work \
  --project naiss2025-xx-yyyy \
  -c pull_timeout_retry.config \
  -resume

Got contacted by the PDC team that the pipeline had launched multiple java processes on the node where the pipeline was started, which triggered the center alert for using too much resources on the login node. Possible cause - CPU demanding local tasks. They were a bit surprised that this happened. I ran it with -profile pdc_kth which should submit the run to slurm and not run anything locally, right? Suggested solution: run via

salloc -p shared --nodes=1 -t 48:00:00

or

sbatch  -p shared --nodes=1 -t 48:00:00

Just wanted to inform the community that this issue came up!

I run nextflow in screen sessions on Dardel often. I normally don't have this issue. However running with

-profile pdc_kth

doesn't submit the entire run to slurm.

nextflow

itself is running on the login node, but the process instances are submitted to slurm. Unless the executor set to

local

, the jobs should automatically select the correct partition and resources, and submit to slurm. It's possible though because of how the pipeline logic is programmed, that

nextflow

itself can take up excessive memory. Similarly if a pipeline uses native processes (

exec

rather than

script

) these run also where

nextflow

is running. Unless you're doing pipeline chaining there's generally no reason multiple instances of the JVM should run. Depending on the pipeline to run, there are various best practices. For a long running pipeline that might take several days, run it on the login node with profile

pdc_kth

in a

screen

session. If it's a shorter running pipeline, that uses a lot of cores, then submitting to a large node and running locally on the node with profile

apptainer

is better practice.

Hi Mahesh. Thanks for the quick reply! I dit it exactly as you said, I ran it from login node in screen with profile

pdc_kth

. That's when the issue came up though. According to the PDC team, 12 core hours were consumed before the pipeline started). I was running the pipeline with 939 samples. Also, in some previous runs, it seems like there was some issues with how temporary files and processes were being handled, so I am setting up scratch manually before running the pipeline (hence the

-work-dir $SCRATCH/nxf_work

flag). Not sure if this is was part of the problem.

Where the work directory is shouldn't make much of a difference. It's all on the same file system ( which is not designed for lots of files at the same time ). It maybe an issue with ampliseq's logic itself, but I haven't looked at what it does. Using 12 core hours before it starts is excessive, however, the filesystem has also been very slow at times because of people poorly using the login node.

alright, thanks! Much appreciated!

I also think 12 corehours sounds excessive, but I also wonder about what they actually said - "had launched multiple java processes" should normally mean separate processes (not threads/lightweight processes of the nextflow monitoring process), and I can't see any good reason why it would do that (as Mahesh mentions, there can be a lot of usage if there is e.g.

exec

usage for processes, or just Groovy code in the workflow or similar - but none of that should appear as separate processes, so that's kind of weird).

hej @pontus, my bad, I completely missed this message and appreciate the feedback. here is what they initially told me: "There is a java process (with many threads) running on login1 right now, here is output from pstree:"

|-screen,1612118,nauras -S ampliseq
  |   `-bash,1612119
  |       `-bash,1624030 run_ampliseq.sh
  |           `-java,1624034 --add-opens=java.base/java.lang=
 ALL-UNNAMED--add-opens=java.base/java.io=ALL-UNNAMED
  |               |-{java},1624294
  |               |-{java},1624298
  |               |-{java},1624299
  |               |-{java},1624300
...

"As I mentioned, it currently seems to be sleeping, probably waiting for the jobs on the compute nodes to finish. When it started, it consumed 12 core hours, which is suboptimal." This was the info I got from the Dardel folks.

Ok, thanks. It seems to indicate that the java process used 12 core hours, which should exclude

local

tasks as well if assume precise communication. Unfortunately, I have a hard time coming up with anything that might use that. Even if there's busy waiting somewhere while resources are being fetched, it should be difficult to get it to that level without it giving up.

All good. Whatever it was, something must have been up, because they contacted me. Anyway, I have started to submit runs via sbatch to avoid this issue! Thanks so much!

One thing that can be said is that you use a new work dir for every run, it might make sense to set

apptainer.cacheDir

in a local config/environment (we've considered it but there's no real good place suitable for everybody). Otherwise, every run will need to pull images again.

Something like this?

# Load required modules
module load nextflow PDC/26.03 apptainer/1.5.1-cpeGNU-26.03

# Set scratch variable

export SCRATCH=/cfs/klemming/scratch/$(whoami | cut -c1)/$USER

# Set project storage variable

export PROJECT=/cfs/klemming/projects/supr/naiss2025-xx-yyyy

# Make directories if they don't already exist

mkdir -p $SCRATCH/nxf_work
mkdir -p $SCRATCH/tmp
mkdir -p $PROJECT/$USER/containers

# Set other variables

export TMPDIR=$SCRATCH/tmp
export SINGULARITY_TMPDIR=$SCRATCH/tmp
export SINGULARITY_CACHEDIR=$PROJECT/$USER/containers
export NXF_SINGULARITY_CACHEDIR=$PROJECT/$USER/containers
export APPTAINER_TMPDIR=$SCRATCH/tmp
export APPTAINER_CACHEDIR=$PROJECT/$USER/containers
export NXF_APPTAINER_CACHEDIR=$PROJECT/$USER/containers

So, e.g.

APPTAINER_CACHEDIR

and

NXF_APPTAINER_CACHEDIR

are different things and we usually don't recommend mixing them.

ok so keep it simple like that maybe?

export SCRATCH=/cfs/klemming/scratch/$(whoami | cut -c1)/$USER
export PROJECT=/cfs/klemming/projects/supr/naiss2025-xx-yyyy

mkdir -p $SCRATCH/nxf_work
mkdir -p $SCRATCH/tmp
mkdir -p $PROJECT/$USER/containers

export TMPDIR=$SCRATCH/tmp
export APPTAINER_TMPDIR=$SCRATCH/tmp

export NXF_SINGULARITY_CACHEDIR=$PROJECT/$USER/containers

Yes. Further review:

whoami

there seems a bit superfluous if you rely expect USER to be set anyway (it should be). I'm not sure if anything cares if you set those `TMPDIR`s.

ok, so nextflow will not need to be pointed to a scratch directory for whatever temporary files will be created? but

nxf_work

probably still good to define for intermediate files that may be needed during runs with

resume

?

Java might make one or two files in

TMPDIR

but that's probably it. Whatever runs belong in scratch or the project directory is a bit of a policy issue.

alright. i was under the impression that it is good practice to place all of these larger intermediate files on scratch where they will get cleaned up from time to time anyway instead of filling up project storage. so even more simplified should do the trick:

# Load required modules

module load nextflow PDC/26.03 apptainer/1.5.1-cpeGNU-26.03

# Set scratch variable

export SCRATCH=/cfs/klemming/scratch/${USER:0:1}/$USER

# Set project storage variable

export PROJECT=/cfs/klemming/projects/supr/naiss2025-xx-yyyy

# Make directories if they don't already exist

mkdir -p $SCRATCH/nxf_work
mkdir -p $PROJECT/$USER/containers

# Set persistent Nextflow container cache

export NXF_SINGULARITY_CACHEDIR=$PROJECT/$USER/containers

Keep track of usage is widely recommended. Depending on project, it might make sense to keep the runs or not. Another consideration is who can access old runs if someone leaves/is sick/parental leave et.c. But something's like that probably good, yes.

alright, that makes sense. however, even though the intermediate files may be gone when i place stuff on scratch, my results should be reproducible when run with the same nextflow and ampliseq version, right? it may take longer obivously because

resume

wont be able to make use of intermediate files, but the same results should be obtained? So after all, should I set any of the things above then, i.e.,

nxf_work

or

NXF_SINGULARITY_CACHEDIR

? Just the latter?

The intention is generally that things are reproducible (and it shouldn't require the same versions). But if it's a non-deterministic pipeline, it wouldn't be.

a follow up question on this: on this page, it is written that the institutional configs (such as applied when running with

-profile pdc_kth

) handle scratch paths. Further down on this page, it is written that

workDir

should point to "fast scratch storage on HPC to keep your home filesystem clean". So it seems to me that there is a good practice recommendation to avoid persistent project storage as the place for temportary/intermediate files?

So, it's good to keep track of different things, there's the general concept of scratch space (be it cluster shared or local) for HPC, but there's also the nextflow

scratch

function. Since dardel is diskless (and things going to local filesystems are completely memorylocked) so far it has not seemed a worthwhile tradeoff to force use of the nextflow

scratch

function there - it does miss small database-like workloads that could likely benefit greatly, but the alternative would be tuning almost all jobs to use rather more memory.

alright thanks, appreciate your input!