https://nf-co.re logo
Join Slack
Powered by
Hello! I was pointed to this channel from <#C0AQYA...
# mag
w
Hello! I was pointed to this channel from #C0AQYAFSSV7 in the sense that there was some interest in a rust rewrite of this pipeline? I have significant Rust experience and way too much confidence that I can probably handle the bio (/hj) (by treating it as somewhat abstract domain knowledge while slowly learning what the words actually mean) - perhaps I could be useful somewhere, starting small?
m
I'd say there is a difference between a tool rewrite and a pipeline rewrite
w
Definitely
I mean, hence the "starting small"
It's more in the sense of "taking a stab at it because I have too much free time and you can throw it out if it's rubbish" rather than "I'll go and handle the whole thing"
m
Oooh
Interesting
w
I dunno, I just like learning through little projects. Maybe there's a relatively small tool somewhere that could be added (I've been gaining an interest in input/output validation, so it's an optional step in the pipeline) or rewritten (for speed and/or correctness)
m
I'll let the pipeline's actual maintainer reply to you here
w
Probably a good idea lol
j
Hi @Werner Kroneman Hrm, so yeah you don't want to re-write the whole pipeline in rust 😉 We would want to identify I guess slow parts of the pipeline that might be more suitable for a rust re-write
Or maybe some abandoned tool...
But I myself do not have time to pair-'develop' so to say such a thing (as I see is what @Phil Ewels rightly suggested)
I duno what the <!subteam^S08MYEKMNTX> thinks, can you think of any particular target tool?
w
Pretty much, yeah. It's picking where to start that I'm stuck on - right now I'm just picking through various Rust bioinfo crates for open issues. I've sent out a PR or two, but still... this would be a slightly deeper dive
👍 1
j
In most cases the suboptimal parts of the pipeline I know would simply be improved by using a more recent tool that already exists....
Maybe some of the binners?
MetaBAT2 and MaxBin2 are popular but basically abandoned for a long itme
w
CI and integration tests would potentially be useful too, or just checking inputs and outputs for weirndess
Huh. Totally could take a look at those two
I'm gonna peek at the repo's and see what's up with them
j
Enjoy sourceforge 😬
j
Hey now, Metabat2 is on Bitbucket 😉
😆 1
w
Let's go click on fake "Install" buttons! :3
😆 1
But yeah, I'll just peek at them and assess their state before doing anything with them
j
MaxBin2 is definitely a pain point in the pipeline - it slows to a halt on large datasets. So a re-write would be far from the worst thing, but I would note that the tool is /old/ and the results it gives are not great in the first place
j
True
Although it is still very popular
w
Let's have an idiot who knows no bio at all try to fix it!
😂 1
What could possibly go wrong...
Heh - write a giant batch of tests, noted :3
j
Like as I understood it, the main advantage of RustQC is that it calculates /a load/ of metrics on a single input file through a single stream through, rather than running 6 tools in parallel. I'm not sure there are many places in MAG where that is the major problem
👍 1
w
Profile and measure before "fixing"
noted :3
Re: MaxBin2, is it alive? As in, you know people who experience the pain and have no alternatives?
What prevents them from switching, currently?
Noticing it has quite a collection of dependencies - using Nix to make those one-command installable would perhaps be useful depending on what state it's in
j
It's a tool that runs a specific algorithm - clustering DNA sequences by coverage (the number of copies in the raw data) and their base composition - to separate out individual genomes in a mixed data file. Other such tools are available, they all operate a bit differently, and some folk prefer one tool or another for their particular data/preferences in ways that mean that offering multiple options can be useful
w
Preserving the algorithms as exactly as possible would be a plus, then. I dunno - definitely a kinda bigger project than I was looking for, but it's still worth at least looking at
So "it's not great, but it probably works best according to some researchers' preferences, and they're trapped in a dying, unmaintained tool"
j
Exactly
w
Seeing a lot of people suffering in the issues
If it's about preserving existing algorithms rather than inventing new ones, it sounds right in my wheelhouse - pretty concrete requirement that doesn't require too much bio knowledge to know if I did it wrong
And like... older algorithms tend to be simpler - perhaps the modern tooling is better, but fails more subtly in ways that are counterproductive in these users' cases
Trying to get the tool running - do you have some input data for me to see what it does?
(if not I'll just go and see if I find some, but it'd be nice if I knew for certain that this tool is known to run on this particular data)
p
All nf-core modules have test data and tests set up
w
Eh, let me just get it to compile
All nf-core modules have test data and tests set up
Thanks, that saves me that possible headache
p
w
Thanks!
👍🏻 1
Yay, it compiles :3
Oh 
115     while (*c == '\t' && *c == ' ' && *c == ',' && *c == ';')
that's a funny line in the original C++
I won't change it so we get bug-for-bug equivalence
🤣 1
looks like quite a bit of dead code too - guessing the original author wanted this to grow further in scope
this is going weirdly smooth so far... this is suspicious
m
I am always suspicious as well when things are too smooth
w
We've seen code go wrong so much, we just distrust it when things just work
🤣 1
Well... hit my session limit, but I honestly got further than I think I had any right to. I wanna run some more verification on it before I link to it, though
On the test data pulled from nf-core the output is exactly identical, I wanna try it with more data now
j
The test data is not suupppper realistic, but that sounds good!
w
Yeah I found something that looks like 300MB of genome data that I might run it on - but that's still only one extra file
Do you have some sample inputs that the tool might realistically encounter in a real pipeline?
d
j
Or maybe cami?
w
Cami was already showing up in search results
d
Yeah, CAMI is the other option
j
https://nf-co.re/mag/5.4.2/results/mag/results-5dabb0159ac0104885e09f301db22126e8fcb394/GenomeBinning/depths/contigs/ If you use the megatest assemblies above, these are the depths inputs you'll need
j
Might not run on a la top
w
Yeah...
j
(either dataset)
w
I don't have access to HPC hardware lol
CAMI had a 300MB dataset, maybe I can get that running, we'll see
The original tool was in C++, so I don't expect the Rust version to be faster, just more maintainable
Well... there's cloud services though. I dunno exactly how heavy these loads are - maybe it's in the realm of being reasonable with what I could do hardware-wise
j
FWIW the really slow experience I had took 8 days and 27.2Gb total memory with 6 CPUs. That was with a really large assembly. So it's not infeasible to do your testing on a laptop, I think
w
Oh that doesn't sound too bad
j
So certainly any improvement on that runtime would be a lovely thing
w
Improving runtime is not a goal yet but it's certainly more feasible with the rewrite
Also all the Perl is gone
Well
j
Also all the Perl is gone
Don't tell my boss
w
One dependency technically uses Perl
Yeah Perl really is the bioinfo language pre-python, isn't it
j
It is... I never encountered it much til I started this job and honestly it is so unreadable
w
Well, keeping my laptop online for 8 days straight sounds annoying... Clearly this thing needs a checkpointing feature
But exact reproduction of the C++ version is the main goal, bugs and all
Python and Rust definitely aren't mutually-exclusive though - could make this into a Python library at some point
There was an open issue in the issue tracker where apparently the old code refused to run parallel when it was supposed to and the user was also sitting there for like... tens of days or something ridiculous
77 days
j
...
better tools /are/ available
w
Yeah
Worth asking how much we wanna, like, actually modify this tool versus keep it in maintenance mode
It totally could be made nicer on some points though
Kinda curious why this 
while (*c == '\t' && *c == ' ' && *c == ',' && *c == ';')
hasn't caused issues yet
j
I would say from my perspective, the value of a reimplementation comes from: 1. fixing obvious bugs - clearly unintended behaviour 2. ensuring good coding practice optimised for improved multithreading/paralellisation for better run time Otherwise, keep it close to the source
w
Probably fair, yeah
Hm
j
my C/++ knowledge is almost 0 so I have no idea what that line is 🥲
w
It checks whether 
*c
is simultanously a tab character, a space, a comma AND a semicolon
That's probably supposed to be OR
j
🫠
w
I keep finding funny lines in bioinfo code like that :3
j
we love TSVs for reasons I'm not entirely sure I fathom, coming from a more statistical background
w
Well good for you, because there was a bug that caused it to emit an extra tab in some of its output files
Don't worry, the Rust also gives an extra tab now
🫢 1
The end-to-end tests are of course what matters for showing it's truly equivalent, but I'm proptesting all the individual functions to see if they're equivalent. I'm cautiously in the belief that this thing actually is equivalent, but it needs more verification
Sadly a bit outta time and braincells to do this today - wednesday will have to be when it happens
🙏 1
Honestly, TSV is fine - tabular data is tabular data. You lose some precision and perhaps eat quite a few more bytes than you should with the binary-to-text conversion, especially on floats, but... not a priority
j
the problem with TSV is using a whitespace character as a separator - many UNIX core tools which we often use to parse large TSV files are rather... whitespace agnostic. Quoting and comma-separating fixes a lot of problems
w
euh
...or maybe some format with a formally-specified parsing grammar 😛
Which quoted csv probably is
j
we always write out JSON these days
w
TSV could be to as long as you're rigorous
I've been learning to love messagepack recently, actually. It's JSON but binary
JSON gets annoying on large datasets if you're repeating the dictionary keys for all the data rows
But... Rust doesn't care, serde is love, serde is life
Open in Slack
PreviousNext
Powered by