A community effort to collect a curated set of analysis pipelines built using Nextflow.

nf-core

I just thought I would quickly post here ~some frustration~ my experiences with `ribodetector` over the last 4 days. In the first instance, this process uses 36 GB of RAM. It continued to fail after raising it 188 GB, and also 388 GB.  After allowing it to use *688 GB* of RAM it then timed out after 8h, 16h, and 24h. (Note: this is a run with 3 samples).

Switching over to rRNA removal using Bowtie 2, all removal processes finished within 40 mins without any issues.

ribodetector has a parameter `--chunk_size 100` which keeps the memory fixed (linear to the number of threads), which makes the memory consumption very manageable without any downside to throughput. If we are not using this parameter in the pipeline, we should. Let me check.

With this parameter enabled, you can ramp up the threads which will reduce the runtime and should avoid the timeout.

That being said, I'm in team bowtie anyway.

That makes sense, it runs without that param and also using `-t 1`.

No worries, I'll add it as a default param

<https://github.com/nf-core/riboseq/pull/142>

I noticed that ribodetector and bowtie2 lack the seed initialization parameters. Shall we add those for deterministic behaviour?

I've added seed in nf-tests where I could, but I'm very strongly of the view that we shouldn't fix seeds in production and give the impression of reproducibility where there is none.

(btw I also came to the conclusion in testing that good ol' Bowtie might be the best way)

I respect that view and we can leave it this way if you like. However, other tools initialize RNG seeds as well (FQ_SUBSAMPLE, STAR_ALIGN). I imagine that most users will want their pipeline to be deterministic, hence my suggestion.

I hear "I want the pipeline to be deterministic" a lot, but I don't think most users understand what that means. If changing the seed meaningfully changes the outcome, that result is inherently unreliable. Fixing the seed just hides the problem. Plus results can still vary across CPU architectures, thread scheduling, and container rebuilds, so "same seed ≠ same results" in practice anyway.

100% with you on all points. It was more a matter of "trying our best to make things deterministic" given that non-deterministic results are a recurring source of issues. Then let's leave it the way it is.

I'm fine with adding default-off options to make things deterministic where possible with health warnings attached. I just don't want folks getting attached to ephemeral results. Also turned off the existing seeds: <https://github.com/nf-core/riboseq/pull/143>

142 good to go if you're OK to approve? <https://github.com/nf-core/riboseq/pull/142>