https://nf-co.re logo
Join Slack
Powered by
Question: what's the state of the CONCOCT binner? ...
# mag
w
Question: what's the state of the CONCOCT binner? It almost looks like it's in a similar state as maxbin2 from my PoV here
d
Yeah, and probably this could be an even better improvement than MaxBin, because it requires running multiple steps, and some of these steps are pure Python. I you can intergrate everything in a single command, that would be great. You can check how we use it right here: https://github.com/nf-core/modules/blob/master/subworkflows/nf-core/fasta_binning_concoct/main.nf
w
Pure python or Numpy? I'd be happy to give it a shot. Is this one truly abandoned or are the authors still active somewhere? I understand I might be kinda stepping on toes by rewriting these tools un-asked, even though technically the license does allow it, and the tools are abandoned.
I'm... still getting used to the politics of it all, to be honest - my goal is not to insult the original authors. I think I'll still carefully give it a shot though - feel free to throw it in the trash if it's not up to standards
And obviously you can be cautious about me actually maintaining it afterwards - I see it as it... more maintainable, so someone forking it will have an easier time than someone forking the original; at least, that's my goal
I'mma be honest... Performance is not my goal, it's just a nice bonus when it occurs - better packaging and maintainability are the main goals while matching the original's output
d
7 years without updates, 70+ open issues... seems dead enough to me: https://github.com/BinPro/CONCOCT/issues?q=sort%3Aupdated-desc+is%3Aissue+is%3Aopen The author still shows some activity on GitHub, though. And don’t worry, no one expects you to maintain this long-term. If you can provide a clean reimplementation with good reproducibility, that’s already a big win. These tools typically don’t need new shiny features, just some fixes sometimes, and that’s much easier with a modern and well-structured codebase.
w
Yeah
I think I'm gonna go for a less radical rewrite for this one... It's calling on probably very highly-optimized Python libraries
What I'll need to rewrite are probably some hot loops that are written in python
👍 1
Gonna try out a new methodology on this one, gonna be fun >:)
https://github.com/werner291/concoct-rs/releases/tag/v0.1.0 I merely dusted it off a bit - builds automatically and reproducibly using Nix, has a Docker image, the tests all run again. All the important core bits are exactly the same. Nice surprise: it's properly deterministic - same seed, same input -> same output
This will be my baseline for the rewrite, but should already be useful as-is with nicer packaging
Hm. The rewrite doesn't so far appear to really give significant performance improvements. The "It's all Python hot loops" seems to not be quite accurate... As far as I can tell, the difficult stuff happens in C, not in Python, and the C so far has not really manifested the same for-free speedups that I observed in maxbin2. CONCOCT authors knew what they were doing, so I'm tempted to abort, and stick to just fixing the packaging
There's some speed-ups left and right with which I could probably bring the runtime down, but they're not optimizations that are inaccessible to the C - at this point I'm not quite sure what the added value of Rust here from a performance PoV
The Nix flake is already seriously bringing down the rate of installation and dependency headaches (down to about 0, really)
Believe me, I want the Rust to be better, but I'm not sure if it's helping the performance much
Open in Slack
PreviousNext
Powered by