Persistent Quota Limit Error `AssocGrpCPURunMinutesLimit` o nf-core #help

*Persistent Quota Limit Error (`AssocGrpCPURunMinu...

Hanna Nordin

12/09/2025, 10:35 AM

Persistent Quota Limit Error (
AssocGrpCPURunMinutesLimit
) on DADA2_TAXONOMY (SLURM/PDC Dardel) Hi everyone, I'm running the nf-core/ampliseq pipeline (

v2.15.0

) on a SLURM-based cluster (Dardel/PDC) using the

singularity

profile. I'm experiencing a critical and persistent issue where all lightweight steps complete successfully, but the heavy process NFCORE_AMPLISEQ:DADA2_TAXONOMY_WF:DADA2_TAXONOMY
fails to start. The job is immediately stuck in PD (Pending) with the reason: (AssocGrpCPURunMinutesLimit)
. This indicates a quota/core-hour limit issue. My situation is slightly complex as my working directory is under project

naiss2025-22-1652

, but my core-hour quota is assigned to project

naiss2025-22-1066

now because I ran out of core-hours in the first project

naiss2025-22-1652

. The main problem is that the DADA2_TAXONOMY sub-job refuses to inherit the correct account (1066). Troubleshooting steps already attempted: 1. Setting
#SBATCH -A naiss2025-22-1066
in the wrapper script for the main Nextflow job. 2. Explicitly forcing the account in `nextflow.config`:

Copy code

process {
    clusterOptions = '--account=naiss2025-22-1066 --time=48:00:00'
}

3. Forcing the account via environment variable (intended to override all defaults):

Copy code

export NXF_CLUSTER_OPTS="--account=naiss2025-22-1066 --time=48:00:00"

The failure persists even with step #3. It appears this specific process is hardcoded or has a default resource request that always gets submitted without the correct

--account

flag to SLURM. Has anyone else encountered a persistent

AssocGrpCPURunMinutesLimit

error with the

DADA2_TAXONOMY

process on a SLURM system, even after trying NXF_CLUSTER_OPTS
? Are there any known workarounds within the nf-core configuration to specifically force the account for this module? Thanks for any advice!

Niklas Schandry

12/09/2025, 10:41 AM

I doubt that this is related to the module (

DADA2_TAXONOMY

). As you noted correctly, this is related to SLURM, and means that you (or your group) ran out of CPU minutes. How many CPU does DADA2 request, and for how long? You can see this by looking at the `.command.sh`in the work directory and checking the

#SBATCH

header. The limit will not allow for jobs to start where the requested CPU minutes are more than whatever you have left on your quota

Niklas Schandry

12/09/2025, 10:44 AM

I kind of assume that whatever 48*60*(number of requested CPU) (=CPUminutes) is larger than whatever your CPURunMinutesLimit is

Niklas Schandry

12/09/2025, 10:45 AM

Since this is

AssocGrpCPURunMinutesLimit

it might only look at the minutes used by currently running jobs.. In any case, probably this is something you need to work out with whoever is managing your SLURM configuration

👍 1

Hanna Nordin

12/09/2025, 11:21 AM

Thank you!

Alexander Nater

12/09/2025, 3:10 PM

It's not quite clear how exactly you are running the pipeline. If you are using PDC Dardel, you should be using

-profile pdc_kth

? The account would then be specified via

--project

Alexander Nater

12/09/2025, 3:12 PM

Or are you using the local executor?

Hanna Nordin

12/09/2025, 4:58 PM

Thanks for the quick feedback! I agree that this may be a SLURM issue related to the CPURunMinutesLimit
, but the core difficulty is that we can't force the job to use the correct quota-rich account (

). To clarify my setup: 1. I am running on Dardel (SLURM executor). I am not using the local executor. 2. I initially ran into issues with the default -profile pdc_kth
and switched to -profile singularity
as a standard workaround to ensure environmental stability (which resolved the issues with the lighter steps). The Crux of the Problem Despite making the correction to use the main billing account (

naiss2025-22-1066

), the

DADA2_TAXONOMY

process is still failing. We used one robust Nextflow method to force the account for sub-jobs:

Copy code

export NXF_CLUSTER_OPTS="--account=naiss2025-22-1066 --time=48:00:00"

Even with this in the S-batch script, the job is still rejected with the quota limit error. So my run_ampliseq.sbatch file looks like this:

Copy code

#!/bin/bash
#SBATCH -A naiss2025-22-1066     
#SBATCH -p shared                
#SBATCH -t 72:00:00              
#SBATCH -n 1                     
#SBATCH --cpus-per-task=16       
#SBATCH --mem=64G                
#SBATCH --job-name=ampliseq_CB   

# Navigate to the correct directory for context
cd /cfs/klemming/projects/supr/naiss2025-22-1652/master_thesis_16S/

# Load environment within the job
module load PDC
module load java/17.0.4
module load singularity

export NXF_CLUSTER_OPTS="--account=naiss2025-22-1066 --time=72:00:00"

# Run the nf-core/ampliseq pipeline
# OBS! Vi använder -profile singularity, INTE pdc_kth,
# för att undvika inbyggda konflikter.
/cfs/klemming/projects/supr/naiss2025-22-1652/master_thesis_16S/nextflow run nf-core/ampliseq \
    -c custom_dardel.config \
    -profile singularity \
    --input SamplesheetDardel.csv \
    --FW_primer "CCTAYGGGRBGCASCAG" \
    --RV_primer "GGACTACNNGGGTATCTAAT" \
    --outdir results/ \
    -r 2.15.0 \
    -resume

My suspicion now is: The

DADA2_TAXONOMY

module's resource request (e.g., 8 CPUs for 48h) is so high that even the correct account (

) might be hitting a concurrent run limit, or the process is still submitting without the correct

--account

flag internally. Since all configuration attempts failed, I have now opened a ticket with NAISS/PDC support to investigate the SLURM configuration for the project quotas. Thanks again for confirming that this is a system/quota issue and not a pipeline bug! Please let me know if you have any more ideas to fix the problem.

Alexander Nater

12/10/2025, 7:59 AM

Sorry, but this is still unclear. If you are running with

-profile singularity

only, you are not using the slurm executor, but local execution on a large slurm allocation. Accounting for the runner job is totally independent from Nextflow. So do you manually change the executor somewhere?

Thomas Danhorn

12/11/2025, 3:24 AM

You are using several different ways of specifying the account (

clusterOption

seems the most reliable to me), and that should work. Have you checked if the account info reaches SLURM? I would first look into any of the

.command.run

files in your work directory to see if it contains

#SBATCH --account ...

. If not, that would be very strange, because

clusterOptions

should put that there. Then I would get one of the SLURM job IDs, either from the

.nextflow.log

, where the lines showing the state of the various processes have a

jobId:

field, or from the process table in the

execuion_report_*.html

, which has a column for that. With that, run

sacct -Po JobID,Account -j <jobId>

, which will tell you the account that was actually used by SLURM. I hope this will shed some light on where the problem lies.

pontus

12/12/2025, 7:45 AM

A few things: • if there's an issue with

pdc_kth

we'd like to fix it to make life easier for everybody, so in that case, please let us know • As Alexander writes, the

singularity

profile alone won't trigger use of

slurm

, so you clearly have configuration you aren't showing - to help troubleshoot it might be helpful showing that • Having to wait is NOT a failure - I haven't checked exactly how PDC does but generally,

AssocGrpCPURunMinutesLimit

wouldn't mean the job will never start but rather that it isn't considered now because of limits - but those typically expire • I don't recognise

NXF_CLUSTER_OPTS

and see no mention of it in the nextflow source - is that something that comes from a LLM perhaps? (Generally, I don't think environment variable settings will override config either) • I don't see how you verify that jobs are indeed submitted on the wrong account (I don't know if you've already run stuff on the new account in which case it doesn't seem unthinkable that you have to wait a bit)

Open in Slack

Previous Next