Hi everyone! I am running ampliseq with DADA2 on a couple of rather large samples and DADA2_ERR is taking a very long time. This is likely because it is using the full 7 x 10^9 bases of the first sample provided instead of the default 10^8 to determine the error models. All my samples have a similar size, so changing the sample used in this step does not solve the issue. As discussed here, DADA2 does not break up samples. It is instead recommended to subsample the reads before running the learnErrors function. Has anyone dealt with this issue before? Is there an easy workaround that I may not have thought of yet? If I implement the subsampling in the pipeline, would there be general interest in including this in the nf-core version or should I just go for a local solution? Cheers, Theresa

I don't remember seeing anyone asking about this issue. Maybe @Daniel Straub has? Generally speaking though: If you implement something generally useful, with tests, we're interested in incorporating it within the official nf-core pipeline. A first thing to do would be to write an issue at the GitHub repo: https://github.com/nf-core/ampliseq/

1e8 bases are specified here, can be overwritten by a config though. Other than that what Daniel said 🙂

Thanks for the fast responses! As far as I understand it, changing the number of bases will not help in my case as the first sample is always used completely: https://github.com/benjjneb/dada2/issues/2054 I might have to have another look at the data to make sure I'm not missing something else, but if the number of bases is really the issue, I'll go ahead and open a GitHub issue.

changing the number of bases will not help in my case

it will speed up the process, but it wont help with sampling from different samples, yes.

I see, thank you! Then something else must be going on in my data. One of my runs is taking significantly longer (i.e. > 24h longer) than the others for the

DADA2_ERR

step. The fact that this run also contains the largest samples would have been an easy explanation.

Just to add to this, there is a way within

learnErrors

to pick a random subsampling of reads across samples:

randomize	
(Optional). Default FALSE. If FALSE, samples are read in the provided order until enough reads are obtained. If TRUE, samples are picked at random from those provided.

Oh thanks @Chris Fields! I forgot that we actually set

randomize = TRUE

including a seed already in version 2.4.0, see https://github.com/nf-core/ampliseq/releases/tag/2.4.0. and this is still true for 2.15.0 (code here and seed here). So I was wrong before, it does make a random draw!

The only thing I’d add here is that we’ve tested a number of variations with

learnErrors

inputs (

nbases

, FASTQ vs dereps objects, and using

randomize = TRUE

vs

randomize = FALSE

). Big memory savings by using FASTQ directly since derep objects have to be read into memory completely, but using dereps elsewhere is definitely worth doing. I also vaguely recall

randomize = TRUE

leading to higher memory usage as well, but would need to verify that, it’s not in our notes 😕