Me again with inode quota concerns! 💾 I can't manage to finish my 240 samples project; I keep busting the scratch file number quotas, which is set to 1M on all supercomputers of the Digital research alliance of Canada. I'm migrating to the only node that allows 10M (Trillium) so it should be fine, but it brought me to question some big picture stuff about /mag. I'm trying to find the main culprits in the pipeline. Pre dastools, I was at 1M so I decide to clear up all .command* files from my work/ (

.command.{err,out,log,trace,run,sh,begin}

, which doesn't break the cache), about 200k files. Resuming, I quickly hit 1M again, with the four Dastools steps generating a lot of files, in my caes for 240 samples * 2 assemblers * 5 binners=2400 dastools jobs. I realise that I could use fewer binners (haven't even used them all!) or just one assembler, but even doing that, a larger project might hit these quotas with more reasonable tool choices (or even space quotas; my scratch stands at 5.6Tb before dastools; mostly bowtie's .bam files fault, but still). Example (work dirs picked at random): RENAME_PREDASTOOL

$ ls work/1e/cf7148e4544a679a1244a47113e4d1/ | head
SPAdes-CONCOCTRefined-761_Jinc.0.fa
SPAdes-CONCOCTRefined-761_Jinc.1.fa
SPAdes-CONCOCTRefined-761_Jinc.10.fa
SPAdes-CONCOCTRefined-761_Jinc.11.fa
SPAdes-CONCOCTRefined-761_Jinc.12.fa
SPAdes-CONCOCTRefined-761_Jinc.13.fa
SPAdes-CONCOCTRefined-761_Jinc.14.fa
SPAdes-CONCOCTRefined-761_Jinc.15.fa
SPAdes-CONCOCTRefined-761_Jinc.16.fa
SPAdes-CONCOCTRefined-761_Jinc.17.fa

$ ls work/1e/cf7148e4544a679a1244a47113e4d1/ | wc
     88      88    3222

DASTOOL_DASTOOL

ls work/03/524ee2d50fb5d91f18f607081c8d16/
SPAdes-DASTool-754_J90.seqlength               SPAdes-DASTool-754_J90_DASTool_summary.tsv  SPAdes-DASTool-754_J90_proteins.faa.archaea.scg         versions.yml
SPAdes-DASTool-754_J90_DASTool.log             SPAdes-DASTool-754_J90_allBins.eval         SPAdes-DASTool-754_J90_proteins.faa.bacteria.scg
SPAdes-DASTool-754_J90_DASTool_bins            SPAdes-DASTool-754_J90_proteins.faa         SPAdes-DASTool-754_J90_proteins.faa.findSCG.b6
SPAdes-DASTool-754_J90_DASTool_contig2bin.tsv  SPAdes-DASTool-754_J90_proteins.faa.all.b6  SPAdes-DASTool-754_J90_proteins.faa.scg.candidates.faa

ls work/03/524ee2d50fb5d91f18f607081c8d16/SPAdes-DASTool-754_J90_DASTool_bins/
SPAdes-COMEBinRefined-754_J90.1077_sub.fa  SPAdes-COMEBinRefined-754_J90.6346.fa     SPAdes-MaxBin2Refined-754_J90.005_sub.fa
SPAdes-COMEBinRefined-754_J90.14186.fa     SPAdes-CONCOCTRefined-754_J90.11_sub.fa   SPAdes-MetaBAT2Refined-754_J90.11.fa
SPAdes-COMEBinRefined-754_J90.2994.fa      SPAdes-CONCOCTRefined-754_J90.22_sub.fa   SPAdes-MetaBAT2Refined-754_J90.1_sub.fa
SPAdes-COMEBinRefined-754_J90.3814.fa      SPAdes-CONCOCTRefined-754_J90.46_sub.fa   SPAdes-MetaBinnerRefined-754_J90.1.fa
SPAdes-COMEBinRefined-754_J90.40.fa        SPAdes-MaxBin2Refined-754_J90.003_sub.fa  unbinned.fa

RENAME_DASTOOLS

$ ls work/f8/9271613dfbf74be1f5a3c7d1aec5f0/
SPAdes-COMEBinRefined-754_J14.0.fa.gz         SPAdes-COMEBinRefined-754_J14.5171.fa.gz  SPAdes-COMEBinRefined-754_J14.973.fa.gz
SPAdes-COMEBinRefined-754_J14.10914.fa.gz     SPAdes-COMEBinRefined-754_J14.5511.fa.gz  SPAdes-CONCOCTRefined-754_J14.21_sub.fa.gz
SPAdes-COMEBinRefined-754_J14.13267.fa.gz     SPAdes-COMEBinRefined-754_J14.7396.fa.gz  SPAdes-CONCOCTRefined-754_J14.70_sub.fa.gz
SPAdes-COMEBinRefined-754_J14.17146.fa.gz     SPAdes-COMEBinRefined-754_J14.8199.fa.gz  SPAdes-DASToolUnbinned-754_J14.fa.gz
SPAdes-COMEBinRefined-754_J14.2769_sub.fa.gz  SPAdes-COMEBinRefined-754_J14.9517.fa.gz  versions.yml

So for a single sample x assembler x binner combination, we have about 170 files produced by Dastools (if we include 4 processes producing 7 .command* files each + versions.yml and .exitcode) where the target output is only 15 fasta files. I've looked at the average over multiple samples and estimate 350,000 files generated by the DASTOOLS processes alone for my 240 samples project. My thinking is that the _"_easy_"_ thing to do (emphasizing the quotes) would be to wrap certain processes (e.g. the four Dastools processes) in a single subworkflow where these four steps are part of a single job; I would argue that having access to the renamed versions of bins we already have elsewhere is a waste of space, so that seems reasonable. In the example above, that would shrink the number of files generated by DASTOOLS in work/ from ~350,000 to ~60,000 assuming that sample is representative; the work dir would only contain what the RENAME_DASTOOLS dir has. Metabinner could benefit from a similar treatment too, but it's really not as impactful as dastools. Other quota gobblers that don't need wrapping, but could have their work/ output cleaned up: • QUAST_BINS, which generates on average 243 files per bin in work/. I'm estimating from my completed jobs (cause pipeline can't complete) that I'll get about 500 bins, which means about 120k files just for QUAST_BINS. • and QUAST, generating on average 22 files per assembly (not as bad). Is there a use for all of this, since nearly none of it is being published to results? Thoughts? holes? questions? Imposter syndrome × my very short experience with nextflow = I might be missing something obvious that justifies this file overload, but here's hoping I'M on to something...

Personally I think this enters in the stage of (maybe) over-optimizing. Unfortunately, of the drawbacks of Nextflow is the intermediate storage and inode usage if you do it in a modular way (one process = one tool). On top of that there are some tools that generate lot of intermediate files and/or heavy files. And finally, sometimes we need those small process exclusively for wiring things up (renaming for example). Start coupling things together In some way would mean moving away from the "modularity and reusability" nf-core principles. For example, doing what you sugest would mean dropping the Das Tool nf-core modules and run our own local module that does everything in one go. That could be justifiable, because 3 of the 4 processes do almost nothing, but that's only one case. If we go for another case, that could mean dropping a few more nf-core modules and implement things locally. And IMHO that's exactly the opposite of what we should be doing; removing local modules by their nf-core counterparts.

I understand. An entry point for bins is also something I thought of ; I didn't mention it, but to get where I'm at, I actually had to restart a clean dir from the assemblies. Having two more entry points would indeed really help and be much more in line with pipeline modularity; plus it would allow cleaning work/ at any of those stages. Does that sound like a hackathon project...? (never been, but will be in Barcelona!)

Yeah, that sounds like a good hackaton project. I personally won't be in Barcelona, but maybe I can make some time to be online at least.

I actually think this is a really important thing, and I've mentioned it many times before as (on my old, small cluster) I used to run into storage and inode problems too. While a certain degree of cruft is to be expected with Nextflow/nf-core, I really do think we can do better at reducing the amount of disk usage with a combination of careful pipeline design, choosing tools that are minimal, and writing modules that are careful to clean up after themselves. In my case, BUSCO puts out both huge numbers of temporary files (which take up lots of space!), so I ended up adding a parameter

--busco_clean

to mag which deletes a bunch of the "working" files that aren't removed. For example here, QUAST as a software is completely overkill for what the pipeline uses it for - we would be better off replacing it with a tool that writes a single file containing summary statistics - bonus points if it can run on multiple input files! In any case, having a bin entry point is something I think would be very valuable; although alternatively I would suggest that MAG should produce bins and stop, and the analysis should be done by another pipeline 😉

I agree that the pipeline work dir of nf-core/mag is excessive. However I do not support compromising on parallelization. more

--*_clean

params might help to a certain degree. Changing software might be also an option, but popular software is not always efficient (but is siught after also in pipelines).

Let's introduce a

--ultra-clean

flag that deletes everything but strictly required files 😅

Yeah core team is aware of that somewhat. It's a big battlr we are having with stopping proposals who want end to end pipelines. But also at this point with mag it's up to you to decide what steps to run and make sure it fits to your infra

For example here, QUAST as a software is completely overkill for what the pipeline uses it for - we would be better off replacing it with a tool that writes a single file containing summary statistics - bonus points if it can run on multiple input files!

Yeah, personally I also think that QUAST is overkill for what we use it. We can look for an alternative (or make one)

VIBE CODED RUST PORT TIME!!

Poor design decisions of the tools that field use can't really be our responsibility

This is true, but as we write the modules /and/ choose the tools, we have a lot of control over both what tool is run and what is retained on disk. And we should be more mindful of doing that, though I would stop far short of /requiring/ that e.g. non-published files are cleaned up.

>

seqkit stats

😉 Not enough Rust on that one, where is the fun? On a serious note, I personally like seqfu more, but yeah, we can definitively evaluate this change to make things lighter on that point

It's a beautiful Go program 🥲

So there seems to be agreement : • QUAST alternative (seqfu, seqkit stats, make it ourselves) → biggest space/inode saver • Additional pipeline entry points for modularity, at least for bins Concerning bin refinement / dastools alternatives, a Binette module was released last week. Is this of interest? (not necessarily as a replacement, but at least as an option)

We already have a volunteer for adding Binette: github.com/nf-core/mag/issues/714

Circling back to this! Any of you attending the hackathon? catalunya I'd like to suggest working on these few things as a project. I've never done this, and being a very new nextflow contributor, I don't want to overstep and I'm not sure if it's my place to direct a project to work on a pipeline I've barely contributed to. So before filling the form, I'd love your input, and to know whether you think these are hackathon-worthy ideas (quast alternative and/or pipeline entry point at bins); or of you can think of any other improvements/features/etc that could be interesting to tackle during the hackathon (I could start a fresh thread in this channel to gather suggestions and ideas)

I am not unfortunately - but don't worry about overstepping :) I'll let the others say if they will work on any stuff

Hey @Jonathan Rondeau-Leclaire! I won't be in Barcelona either, and I will try to be online, but can't promise anything, specially considering the 5 hours of difference. But we can at least make a plan to work on those improvements and maybe other ones and coordinate the efforts on the development.