Dear all, We*** are about to embark on a new round of adding functionality, fixes and improvements for our favourite Ribo-seq workflow - do expect some activity here and/or on Github soon, and please engage as much as possible to get the best possible outcome for everyone (* @Jonathan Manning, @Naiara, @Sebastian Uhrig). In an attempt to enable the use of ORFs for downstream analysis (instead of just canonical genes quantified by Salmon), we set out run a benchmark of 6 different ORF callers, namely

RiboCode, Ribo-TISH, Ribotricer

(already in Ribo-seq), as well as

PRICE

,

Rp-Bp

and

RibORF

(not (yet) in Ribo-seq). The additional tools

RIBOSS

or

Rp3

were excluded from the benchmark as they had additional requirements. Please see the attached benchmark report below, happy to hear your experiences and feedback. As a quick note: the tools for this run were hand-selected by @Naiara and several of them had been tested (manually) on the dataset before (a dataset of good quality with credible biological outcome, 2 groups, 3 technical replicates each). The engineering and orchestration was aided by jean claude code van damme . Here are the main takeways: • Tier-1 cohort (recommended production defaults):

RiboCode

(mean Spearman 0.917, 6-26 min/rep, biology-correct uORF/dORF 1.7-4.0×) and

Rp-Bp

(mean Spearman 0.893, ~20-24 h/rep, biology-correct uORF/dORF 3.4-5.2×). Both tools have non-overlapping personalities:

RiboCode

is a permissive canonical-CDS caller;

Rp-Bp

is Bayesian-strict. • *Tier-2 cohort (opt-in specialists): `Ribo-TISH`*(alternative-start-codon discovery within annotated CDSes; calls ~60% Truncated alt-starts) and

PRICE

(near-cognate start-codon detection; only 29.7% AUG vs 70% non-AUG CUG/GUG/AAG/UUG/AUU/AGG/ACG/AUA — unique non-overlapping niche). • Excluded:

ribotricer

(mean Spearman 0.288 — rank-unstable scoring across replicates makes its score column unusable for production);

RibORF

(is operationally untenable at genome-wide scale — readDist.pl re-parses the SAM file once per read-length class, projected 5-7 days per replicate on this cluster’s hardware);

RIBOSS

(missing RNA-seq dependency);

Rp3

(missing proteomics dependency). PS: Slack appears to block all attachments currently (issue: https://slack-status.com/2026-05/fe557ca05fdb64fa), I’ll retry once this technical issue is fixed.

Here's the project board: https://github.com/orgs/nf-core/projects/155

Here is the report:

this is excellent work, thank you for sharing! Are you planning to extend the benchmarking to a few more datasets? I wonder how these results vary with dataset quality 🤔 And I was curious whether ORF-RATER was considered as a candidate - tho in my experience this tool is difficult to install/set-up..? either way, do keep us posted and I'll try to help where I can as well

Thanks Ira! It was currently not planned to add additional datasets for the benchmark - one could arguably add some bad data as well 🙂 If you have a specific interest I could potentially consider adding ORF-RATER to the benchmark?

I am mentioning ORF-RATER because of this interesting paper https://www.science.org/doi/epdf/10.1126/science.aay0262 and I get the impression is generally a well-respected tool - no personal interest. In terms of data quality - was thinking the opposite, to include an extreme periodicity dataset ... expectation here being that tool performance would increase across the board and agreement should improve. And if some ORF callers are terrible even with best quality datasets...well that's useful information. I think even trying an independent dataset of similar quality might help us be sure results aren't dataset specific and are generalizable. But prolly for the aim here which primarily seems to be prioritising tools for implementation this isn't absolutely necessary.

I think the idea of ORF-RATER is solid but its just a repo of scripts and my intuition is that semi-decent packaging should be a prerequisite for inclusion in nf-core/riboseq. Realistically that was why ribotricer made it in even though we knew it was poor from the outset. I do think this is a domain that would benefit from getting RustQC'd (if thats a verb now). There are plenty of good ideas but most are poorly implemented and running multiple does a tonne of redundant work. I have kind of done this already in python where I re-implement the methods/metrics where I can

Rustification is awesome in some ways, and I totally agree that the riboseq field is perfect for it- tooling is highly specialist and therefore sparsely used, and therefore a little lacking in the engineering department 😉 . But we do need to consider ownership and maintenance. It's no good Rustifying a tool only to abandon it. Practically, that probably means buy-in from tool authors

oh wow, someone had a busy weekend!

Do you reckon there is no place for going from a poorly written, non-performant, unmaintained codebase to a well-written, performant, unmaintained codebase? Its incredible what you are managing to get done with Claude @Jonathan Manning. I have half a mind to get Codex to analyse your commit history on that draft and see what I can learn. Out of curiosity, do you just have a Max subscription and do you ever hit token limits on that? Any tricks you'd recommend to keep token usage down?

Tbh I don't even want to think about the amount of tokens that took! We do have the all-you-can-eat plans here, though I suspect those won't be a thing much longer. But I figure my time is expensive as well... The real power is hooking it up to testing so it can bugfix and iterate. So I had it running full-sized tests via Platform.

hastily-written-then-abandoned rewrite

I think that's the crux. Is the rewrite-r (RustQC-er?) willing to stick around and keep improving it?

I think it's highly unlikely they will, just as a scaling issue. Folks can accumulate rewrites at a much higher scale than their ability to maintain them all.

I do think stewardship is important and broadly agree with how it is included in rewrites.bio. But I also suspect that we will inevitably end up with a class of software analogous to thesisware that are rewrites that are genuinely useful improvements, but which never acquire long-term institutional maintenance in the same way most academic software doesnt. That is of course not ideal. But I’m not convinced that “future stewardship cannot be guaranteed” is, on its own, sufficient reason not to do a rewrite especially in these contexts where it is the method that is the scientific product rather than the software that implements it. Interesting times and each of your leadership has been massively influential so I don't want to overlook that just also feel there is an opportunity to have better frameworks for this middle ground software too. Not just in rewrites but thesisware more broadly

Probably we need community orgs that adopt software for community maintenance. e.g. the latest STAR rewrite is in 'scverse'. Then at least we don't end up with things stranded in people's personal GitHub accounts where nobody else can touch them.

Yeah, I've had this in the back of my mind for a while now. An nf-core for rewrites, of sorts. Needs a lot of caution and work to get it right though, that path is an indiana-jones esque path of dangers 😅

It really is treacherous haha the more I think about it the more potential issues I come up with rather than solutions..

image.png

Don't get me wrong, I'm happy to have all the fun of the rewrite. I just don't want to own them after 😄

You're the perfect collaborator really @Jonathan Manning - the auditor at the end of the path 😆

Good evening! Joining the thread a bit late, but I wanted to document our tool selection logic here for the record. We benchmarked five tools representing distinct algorithmic approaches: Ribotricer (multiscan phase-score periodicity), Ribotish (Negative Binomial testing of initiation peaks), RiboCode (combined P-site periodicity and TIS detection), PRICE (EM-based probabilistic modeling of sub-codon footprints), and Rp-Bp (Bayesian MCMC periodicity). These were vetted against two production criteria: active maintenance (recent commits and containerized installation), and methodological diversity. Several other established tools were excluded after initial review: • RibORF: RibORF was evaluated based on its widespread adoption in the field; however, it was ultimately deprioritized due to a combination of poor performance in our early trials and a lack of active software support. • RIBOSS: Omitted because its framework requires paired RNA-seq data, which is outside our current pipeline’s scope. • ORF-RATER: I agree with Ira that ORF-RATER is highly regarded and that its regression-based approach to overlapping ORFs offers a unique methodological perspective. However, as previously noted, its Python 2 lineage and the lack of meaningful updates since 2020 mean it does not align with the standards we "established" for this benchmark. That said, I’m happy to revisit the list if anyone feels a key tool is missing or if any of our current choices seem redundant...

Oof, Python 2 and unmaintained? Ok then that's rewrite gold..

Do you already have plans for the weekend, Phil?

I think it's more @James Ferguson you should look to now! 😂

Name mentioned. What am i doing now?

You wanted more reasons to spend all those sweet, sweet tokens you have, right? 😆 I'm just going to tag you every time someone asks me to do a rewrite from now on.. homer disappear

I mean, i can re-write them, then transfer the repo to whoever wants to maintain it lol

Yeah the idea isn't new, we've bounced it around within Seqera for years, even in pre-LLM times (aka "the orphanage for abandoned bioinformatics tools")

Someone needs to pay for it in the end. No matter how many people volunteer