I was trying to run ampliseq on the PDC from KTH for the first time (I used to run it on Uppmax before without problems). I used the following: Version nextflow/24.04.2 Singularity: export NXF_SINGULARITY_CACHEDIR=/path/to/my/dir/ Params file: project : "naiss2024-myproject" input : "samples.tsv" outdir : "results" FW_primer : "NNNNCCTACGGGNGGCWGCAG" RV_primer : "GACTACHVGGGTATCTAATCC" trunclenf : 269 trunclenr : 209 double_primer : true skip_qiime : true sample_inference : "independent" dada_ref_taxonomy : "sbdi-gtdb" run code: nextflow run nf-core/ampliseq -r 2.12.0 -profile pdc_kth -params-file params.yml It worked till it stopped at the step “NFC…DA2_TAXONOMY_WF:DADA2_TAXONOMY”. Then I checked the JobInfo and got the code “AssocGrpCPURunMinutesLimit”. With help from the support I figured out that it means: “the job was sent to the long partition, with a time limit of 1.5 days (36 hours). The long partition is an exclusive partition, meaning that the job avails the entire node exclusively, and is charged for the same, irrespective of how many cores are actually used. Based on this, this job would account for 36*128 = 4608 core-hours, while my monthly allocation is 2000 core-hours.” Any idea how I can avoid this? What settings should I change? Thanks for the help :)

You can see in the

pdc_kth

profile, how the partition a job is submitted to is chosen:

if (time <= 7.d && mem <= 111.GB && cpus <= 256) {
                extra += ' -p shared '
            }
            else if (time < 1.d) {
                // Shortish
                if (mem > 222.GB) {
                    extra += ' -p memory,main '
                } else {
                    extra += ' -p main '
                }
            } else {
                // Not shortish
                if (mem > 222.GB) {
                    extra += ' -p memory '
                } else {
                    extra += ' -p long '
                }
            }

So in order to be submitted to the

long

partition, your job must have requested more than 222GB of memory for a runtime of more than 1 day. I am not familiar with the #CEA7TBJGJ pipeline or this particular tool, but both seem quite excessive to me. If you still have logs from old runs in Uppmax, you could check how much the tool truly needed in the past and adapt your configuration accordingly.

One way is of course to get a higher allocation. A better alternative is probably to tune the process (https://nf-co.re/docs/usage/getting_started/configuration#tuning-workflow-resources).

The next release of Ampliseq (2.13.0; I thought it was released yesterday but it doesn't seem it's out yet) has the possibility of partitioning the DADA2 taxonomy step to reduce memory requirements. If you have a lot of ASVs, this could help you a lot since the this tends to be the most memory consuming step and the tool is written in what seems a non-optimal way and therefore takes more and more memory. The parameter to control this is

--dada_assign_chunksize

, set to 10,000 by default. How many ASVs do you have?

Ah ok, maybe worth to wait till its out then. I have overall 66 samples of which in total i have 11.489.336 reads (174.080 average). Havent checked the unique ASVs yet.

Two things: 1) The number of ASVs is what decides this. Easy to find out:

wc -l results/dada2/ASV_table.tsv

. 2) You will likely still have to configure resources, but less. I haven't run Ampliseq at Dardel yet, but at Rackham I had a

nextflow.config

like this:

process {
  withName: DADA2_ADDSPECIES {
    memory = 110.GB
    cpus = 10
    time = '6d'
  }
  withName: DADA2_TAXONOMY {
    memory = 110.GB
    cpus = 10
    time = '6d'
  }
}

There are two problems with the above. I would reduce the time to maybe

'1d'

(to avoid getting into the "long" queue) and increase the number of cpus. The way I've understood how cpu allocation works for Nextflow at Dardel is that the

cpus

in a Nextflow config translates to "virtual cpus" (two hyperthreads per physical core) and there 256 virtual cpus for each normal node. Since there's 256 GiB RAM, of which 220-240 or so is available to user processes, for each GiB RAM you ask for you should ask for a cpu. If you ask for less your Nextflow process will only see that smaller number. (Not so important, I believe, for DADA2 taxonomy, but as a general principle.) So when you allocate 110 GiB RAM, ask for, say, 100 cpus too (less than the memory to be on the safe side of not allocating more cpus than required as they're not going to be put to good use anyway). @Matthias Zepper and @pontus: Is this in line with how you understand this? With 2.13.0 of Ampliseq you can probably reduce the memory requirement. In my runs I've had between ~700 and 65,000 ASVs, so maybe accepting the default 10,000 per batch and opting for 64 GiB memory could work. Please post what you find to work here. Edit: I don't actually think the

DADA2_ADDSPECIES

needs the same amount of resources as

DADA2_TAXONOMY

, so try to set that lower. At Dardel economizing on cpu cores is much more important than at Rackham due to the incredible number of cores per node.

My recommendation is to set resources according to your process - as you mention, asking for a lower number of cpus than the appropriate number from the memory share will typically restrict you more than asking for the "correct" number, but most bioinformatics code doesn't scale to those levels anyway, but again, tune to your actual use (and use

resourceLimits

for e.g. supporting systems with fewer cores). I never recall its setup but dardel seems to have two sockets of 64 core CPUs that are exposed with hyperthreading (or whatever AMD calls it) as 256 cores in total (others do other stuff with hyperthreading, but I guess this is what cray felt like a good thing).

Hi, thanks for the help, the nextflow.config adding with ressourceLimits worked, at least for the first taxa annotation but then I got the same problem for the addSpecies again. I guess I need to play a bit with the CPU and memory limits. I am not so familiar with this. (Or I try the process limitation for the specific steps example from Daniel). But thanks for the help! So I know what to do. :)

The

pdc_kth

should set appropriate

resourceLimits

, what you need to do is to provide good values for the different nextflow processes (see the link posted above for additional information).