https://nf-co.re logo
Join Slack
Powered by
HI all, I’m in the midst of a rewrites.bio-influen...
# ampliseq
c
HI all, I’m in the midst of a rewrites.bio-influenced implementation of DADA2 in Rust, which currently seems to be ~5x faster using 8x less memory on the Illumina MiSeq SOP data (not including R startup cost). I’ll be going through and validating results over the next week or two, but the work is here: https://github.com/HPCBio/dada2-rs
🎉 1
Will still need some fine tuning and testing over the next week or two; most ‘message passing’ between functions is using simple JSON which could be switched out for something else. But most functionality has been ported.
(Benchmarks are on the learn-errors step)
d
Thats interesting. The pipeline is with the usual MiSeq run not very resource hungry compared to other pipelines, but for larger projects that can really make a difference I think.
c
Yes, I’m cleaning up the code at the moment to catch out any other odd issues, dealing with lint and clippy suggestions. And I’m running comparisons to DADA2's final output. But so far looks promising.
Just a small update on this (might be of interest to @Phil Ewels): I checked in w/ Ben Callahan, the main DADA2 developer, and he seems to be completely onboard with a project like this based on rewrites.bio.
p
Great!
What's the status of the original codebase? Could this be a v2 of the original tool, as done for TrimGalore? (Best case situation in my opinion)
c
@Phil Ewels for now we’re operating as if it’s a separate port. A few advantages to this. The basic message-passing between steps uses JSON, so it’s conceivable an updated R release could simply wrap this, or someone could implement a similar wrapper in Python.
The main reason it was implemented in R/C++ was one step where one would use a model to learn error rates, which is used in the actual denoising step. This used R-based functions, but we kinda open this up to pure Rust or external calls: https://github.com/HPCBio/dada2-rs/tree/main/examples/external_errfun
So we can test R-based calls like the original version, use Python-based calls, etc. Probably dangerous as heck
But we can at least test vs the original R models for consistency
p
Nice! Maybe you know it already but there's also extendr for rust in R: https://extendr.rs/
Then could have it as a rust CLI, but also publish the internal functions as a rust library. Then have the r package load those functions and just execute them as a wrapper
Depends on the data volume etc, but sounds like passing intermediates via JSON would be comparatively slow / a bottleneck?
But I'm not clear on what DADA2 actually does 😅
c
Yep extendr is one avenue I’m looking into, and it may be a bit easier than an external callout.
👍🏻 1
The various DADA2 functions are essentially a mini-pipeline. A number of modes in the central denoising step DADA2 implements can be run essentially map/reduce, so instead of running serially can be fanned out and then collected if needed.
JSON helps but is prob a short-term solution to pass data across different stages in the DADA2 workflow. It was just easier to get up and running. I’ve thought about other options but I’m completely open to ideas there
👍🏻 1
p
Probably depends on what the data looks like, but Parquet is what all the cool kids seem to be using these days 👍🏻
c
Yep, that’s a possible option. Avro was another I’ve heard about
👍🏻 1
Open in Slack
PreviousNext
Powered by