I have a little thing to discuss: Currently we have

--min_contig_size

, but it doesn't work completely as expected. Not every binner supports it, so in practice it is only enforced for MetaBAT2, SemiBin2 and MetaBinner. MaxBin2 falls back to its internal 1000 bp default because we never pass

-min_contig_length

, CONCOCT has no length filter at all and chunks every contig, and COMEBin has no way to set one either (issue reported: github.com/nf-core/mag/issues/1100). So the binners are not seeing the same contigs, which makes them hard to compare and wastes compute on contigs we are going to discard anyway. The cleaner solution is to filter the assemblies before feeding them to the binners. There are two options on where, because the read alignment is shared: 1. Right after assembly. The filter applies to everything: binners, assembly QC and the ancient DNA path. Simplest to implement, no BAM handling. Two drawbacks: ancient DNA loses the short fragments, and ALE and DeepMAsED end up evaluating a trimmed assembly. It also shifts the coverage values, since reads that used to map to short contigs get reassigned or go unmapped. 2. Map the full assembly, filter afterwards. Assembly QC and ancient DNA keep everything, and the depths stay exactly as they are today. The cost is that we have to filter the BAMs too, one filter per assembly plus one per BAM. Filtering only the FASTA and leaving the BAMs untouched is not an option, I tested it: COMEBin dies with a

KeyError

in

gen_cov.py

and MetaBinner with an

IndexError

in

component_binning.py

, both because a reference present in the BAM is missing from the FASTA. MetaBAT2, MaxBin2, CONCOCT and SemiBin2 tolerate it and return identical bins either way. Thoughts <!subteam^S08MYEKMNTX>?

I ran into this problem with our metagenome pipeline. I opted for option 2, as storage space is not limiting for us. But currently our pipeline is single-sample. Most importantly with option 2, you can map reads to circular contigs before removing them from the assembly for binning, so you have coverage for those genomes while not binning them. For MAG, given the potential scale problem (we definitely have users who run with a lot of data!) I would be wary of duplicating every BAM on disk. And there are questions to decide like how to filter a BAM - if you remove a contig, and have one read that mapped to it while its mate mapped to a contig that's remaining, do you drop both reads from the BAM?

Yeah, definitively duplicating the BAMs can be an issue in mag I guess. Regarding how to handle paired reads, I don't think it's a big deal, it's equivalent to what already happens today when one read maps and it mate doesn't. On paper the drawbacks for option 1 look bigger though, because they affect the results. But to be fair, I'm not sure how many people really use the assembly qc metrics, I've also been thinking if we should make ALE opt-in in that sense. And aDNA is something very specific too. @James Fellows Yates could you give us an idea how bad would be option 1 for aDNA, considering that our default value is 1500? We can always compensate it with documentation and even a warning if that's enough.

Another option would be to try and fix COMEBin/MetaBinner? Certainly the COMEBin author is presently active and could maybe be persuaded to issue a fix?

Yeah, that's an option I haven't considered. We can try that route first.

I think we should be willing to discard tools that are old, unmaintained and give bad results! 😜

Personally I think we can consider it. We can exchange it for something better, like VAMB.

Sorry I missed this!

@Diego Alvarez Saravia @Jim Downie for ancient DNA samples, I usually prefer to keep all contigs longer than 500 bp and align the short-read data against them because we have so many short contigs due to highly fragmented input data and would discard otherwise up to 80% of the contigs. These contigs cannot really be used for any binning algorithm, but these are useful for some reference-based binning approaches. Therefore, I would argue that option 2 is the much better one. I don't complete understand why option 2 would duplicate the sequencing data but in the

work

directory, but maybe I am missing something.

Hey @Alex Hübner, thanks for your insights! Ok, then I will try to go with option 2 for this if we don't get the COMEBin PR merged. The duplication comes from filtering the BAM (the original one will be there already, and we'll get a filtered one with probably a similar size).

Thanks @Alex Hübner!

I was just about to post about this as my deeply sequenced environmental sets are essentially impossible to bin without filtering the assemblies ( 17 million plus contigs in some samples). I had tried the --min_contig_size flag and was looking at the code as saw there was no actual filtering step. Glad to see a solution is being discussed, for now I am just using seqkit and re-running from scratch on assemblies/cleaned reads from the prior run.

Yeah looks like people are getting too good at metagenomic sequencing and more regularly getting larger and larger sequencing datasets... I guess something that should be prioritised..

Just want to mention that I hit a BAM limit in github.com/nf-core/mag/issues/267, so filtering a metagenome before mapping might be also helpful with large data.

Well, I didn't get a response to the COMEBin PR, so I guess we'll have to roll out the filter. To mitigate the main negative effect of implementing this (i.e. the double mapping) I'm thinking on making ALE opt-in, so by default users won't have to pay extra storage, unless they want full assembly QC statistics (ALE and/or DeepMAsED) or aDNA correction.

I think that makes sense. I don't think ALE is super popular Vs QUAST for example

I was going to add, ALE also generates ~700gb + outputs with novaseq depth metagenomic runs of complex communities.

@Derek Smith we already handled that on 5.5.0 (full output disabled by default)

Ahh nice, I was wondering why the heck my output and work folders were like 20 Tb in size until I figured that out haha.

PR opened to address this: github.com/nf-core/mag/pull/1118