Hi everyone!! I’m processing 538 paired-end 16S amplicon samples (~200 GB total) with the

--sample_inference pooled

option enabled for DADA2. However, the run failed with the following error:

Error executing process > 'NFCORE_AMPLISEQ:AMPLISEQ:DADA2_DENOISING (1)'

Caused by:
  Process exceeded running time limit (2d)

I’m using the default configuration (6 threads, 36 GB memory) per task on our HPC. I suspect this might be related to memory limitations, given the dataset size and pooled inference mode. Does anyone have suggestions on how to estimate or scale memory and CPU requirements for large datasets like this (~500 samples, 200 GB)? Would it make sense to split the analysis into batches or switch to the default mode instead? Any advice or shared experience would be greatly appreciated! 🙏

Hi there,

Would it make sense to split the analysis into batches or switch to the default mode instead?

Indeed, default is much faster, `--sample_inference pooled`scales badly with increasing sample size (quadratic scaling):

Independent sample processing has two major advantages: Computation time is linear in the number of samples, and memory requirements are flat with the number of samples.

from https://benjjneb.github.io/dada2/pool.html

Is there a way to increase sensitivity to rare variants by sharing data across samples without losing the fast linear-scaling of computation time with dataset size? Our idea is to implement “pseudo-pooling”

from https://benjjneb.github.io/dada2/pseudo.html#pseudo-pooling So for such a large dataset I would advice for

--sample_inference

values

independent

or

pseudo

, but not

pooled

.

Does anyone have suggestions on how to estimate or scale memory and CPU requirements for large datasets like this (~500 samples, 200 GB)?

No, afaik there is no systematic evaluation (yet?) about sample number, sequencing depth, sample complexity to computational requirements.

Thank you for the valuable suggestions @Daniel Straub !!

Agree w/ @Daniel Straub there isn’t a hard and fast rule. More sequences, more samples, and longer reads (PacBio vs Illumina) will increase time, but the hardest to assess a priori is how complex/diverse the microbiome is per sample (let alone across a whole set of samples).

Have you checked the output in /work/task/hash ? If it's a memory problem the system will send a message and you will find it in the hidden files. Also if you have a job ID you can use

seff jobID

in HPCs with slurm to instpect the amount of resources used