Dear #CE9MS66BS people, what do I need to do to put mag in #C024915L4KB cf https://github.com/nf-core/fetchngs/issues/376

Me again with inode quota concerns! 💾 I can't manage to finish my 240 samples project; I keep busting the scratch file number quotas, which is set to 1M on all supercomputers of the Digital research alliance of Canada. I'm migrating to the only node that allows 10M (Trillium) so it should be fine, but it brought me to question some big picture stuff about /mag. I'm trying to find the main culprits in the pipeline. Pre dastools, I was at 1M so I decide to clear up all .command* files from my work/ (

.command.{err,out,log,trace,run,sh,begin}

, which doesn't break the cache), about 200k files. Resuming, I quickly hit 1M again, with the four Dastools steps generating a lot of files, in my caes for 240 samples * 2 assemblers * 5 binners=2400 dastools jobs. I realise that I could use fewer binners (haven't even used them all!) or just one assembler, but even doing that, a larger project might hit these quotas with more reasonable tool choices (or even space quotas; my scratch stands at 5.6Tb before dastools; mostly bowtie's .bam files fault, but still). Example (work dirs picked at random): RENAME_PREDASTOOL

$ ls work/1e/cf7148e4544a679a1244a47113e4d1/ | head
SPAdes-CONCOCTRefined-761_Jinc.0.fa
SPAdes-CONCOCTRefined-761_Jinc.1.fa
SPAdes-CONCOCTRefined-761_Jinc.10.fa
SPAdes-CONCOCTRefined-761_Jinc.11.fa
SPAdes-CONCOCTRefined-761_Jinc.12.fa
SPAdes-CONCOCTRefined-761_Jinc.13.fa
SPAdes-CONCOCTRefined-761_Jinc.14.fa
SPAdes-CONCOCTRefined-761_Jinc.15.fa
SPAdes-CONCOCTRefined-761_Jinc.16.fa
SPAdes-CONCOCTRefined-761_Jinc.17.fa

$ ls work/1e/cf7148e4544a679a1244a47113e4d1/ | wc
     88      88    3222

DASTOOL_DASTOOL

ls work/03/524ee2d50fb5d91f18f607081c8d16/
SPAdes-DASTool-754_J90.seqlength               SPAdes-DASTool-754_J90_DASTool_summary.tsv  SPAdes-DASTool-754_J90_proteins.faa.archaea.scg         versions.yml
SPAdes-DASTool-754_J90_DASTool.log             SPAdes-DASTool-754_J90_allBins.eval         SPAdes-DASTool-754_J90_proteins.faa.bacteria.scg
SPAdes-DASTool-754_J90_DASTool_bins            SPAdes-DASTool-754_J90_proteins.faa         SPAdes-DASTool-754_J90_proteins.faa.findSCG.b6
SPAdes-DASTool-754_J90_DASTool_contig2bin.tsv  SPAdes-DASTool-754_J90_proteins.faa.all.b6  SPAdes-DASTool-754_J90_proteins.faa.scg.candidates.faa

ls work/03/524ee2d50fb5d91f18f607081c8d16/SPAdes-DASTool-754_J90_DASTool_bins/
SPAdes-COMEBinRefined-754_J90.1077_sub.fa  SPAdes-COMEBinRefined-754_J90.6346.fa     SPAdes-MaxBin2Refined-754_J90.005_sub.fa
SPAdes-COMEBinRefined-754_J90.14186.fa     SPAdes-CONCOCTRefined-754_J90.11_sub.fa   SPAdes-MetaBAT2Refined-754_J90.11.fa
SPAdes-COMEBinRefined-754_J90.2994.fa      SPAdes-CONCOCTRefined-754_J90.22_sub.fa   SPAdes-MetaBAT2Refined-754_J90.1_sub.fa
SPAdes-COMEBinRefined-754_J90.3814.fa      SPAdes-CONCOCTRefined-754_J90.46_sub.fa   SPAdes-MetaBinnerRefined-754_J90.1.fa
SPAdes-COMEBinRefined-754_J90.40.fa        SPAdes-MaxBin2Refined-754_J90.003_sub.fa  unbinned.fa

RENAME_DASTOOLS

$ ls work/f8/9271613dfbf74be1f5a3c7d1aec5f0/
SPAdes-COMEBinRefined-754_J14.0.fa.gz         SPAdes-COMEBinRefined-754_J14.5171.fa.gz  SPAdes-COMEBinRefined-754_J14.973.fa.gz
SPAdes-COMEBinRefined-754_J14.10914.fa.gz     SPAdes-COMEBinRefined-754_J14.5511.fa.gz  SPAdes-CONCOCTRefined-754_J14.21_sub.fa.gz
SPAdes-COMEBinRefined-754_J14.13267.fa.gz     SPAdes-COMEBinRefined-754_J14.7396.fa.gz  SPAdes-CONCOCTRefined-754_J14.70_sub.fa.gz
SPAdes-COMEBinRefined-754_J14.17146.fa.gz     SPAdes-COMEBinRefined-754_J14.8199.fa.gz  SPAdes-DASToolUnbinned-754_J14.fa.gz
SPAdes-COMEBinRefined-754_J14.2769_sub.fa.gz  SPAdes-COMEBinRefined-754_J14.9517.fa.gz  versions.yml

So for a single sample x assembler x binner combination, we have about 170 files produced by Dastools (if we include 4 processes producing 7 .command* files each + versions.yml and .exitcode) where the target output is only 15 fasta files. I've looked at the average over multiple samples and estimate 350,000 files generated by the DASTOOLS processes alone for my 240 samples project. My thinking is that the _"_easy_"_ thing to do (emphasizing the quotes) would be to wrap certain processes (e.g. the four Dastools processes) in a single subworkflow where these four steps are part of a single job; I would argue that having access to the renamed versions of bins we already have elsewhere is a waste of space, so that seems reasonable. In the example above, that would shrink the number of files generated by DASTOOLS in work/ from ~350,000 to ~60,000 assuming that sample is representative; the work dir would only contain what the RENAME_DASTOOLS dir has. Metabinner could benefit from a similar treatment too, but it's really not as impactful as dastools. Other quota gobblers that don't need wrapping, but could have their work/ output cleaned up: • QUAST_BINS, which generates on average 243 files per bin in work/. I'm estimating from my completed jobs (cause pipeline can't complete) that I'll get about 500 bins, which means about 120k files just for QUAST_BINS. • and QUAST, generating on average 22 files per assembly (not as bad). Is there a use for all of this, since nearly none of it is being published to results? Thoughts? holes? questions? Imposter syndrome × my very short experience with nextflow = I might be missing something obvious that justifies this file overload, but here's hoping I'M on to something...

github.com/ziyewang/COMEBin/…/1.1.0 🎉 Fixes some of the pain points and exposes some more parameters (no size filter for contigs though still).

Hello! I put this message in #CE6SDBX2A earlier and was kindly directed to this channel. If anyone could help me with this issue I have been having with phydamage not exiting I would greatly appreciate it!

Hello! I am using nf-core/mag with nanopore long-read data. I have a question about minimum read lengths. I see in the pipeline QC parameters that the --longreads_min_length is set to default: 1000. Does this meant that reads <1000 bases are skipped just for the QC steps, or are reads <1000 skipped for all the following steps in the pipeline? thank you!

We had a discussion of whether to and, primarily how to, implement dereplication: github.com/nf-core/mag/issues/413. I think I will have time quite soon to start on this, so if we could settle on the implementation would be good. (GitHub is horribly slow today; I can't open the issue but am relying on Claude's memory.)

I have a little thing to discuss: Currently we have

--min_contig_size

, but it doesn't work completely as expected. Not every binner supports it, so in practice it is only enforced for MetaBAT2, SemiBin2 and MetaBinner. MaxBin2 falls back to its internal 1000 bp default because we never pass

-min_contig_length

, CONCOCT has no length filter at all and chunks every contig, and COMEBin has no way to set one either (issue reported: github.com/nf-core/mag/issues/1100). So the binners are not seeing the same contigs, which makes them hard to compare and wastes compute on contigs we are going to discard anyway. The cleaner solution is to filter the assemblies before feeding them to the binners. There are two options on where, because the read alignment is shared: 1. Right after assembly. The filter applies to everything: binners, assembly QC and the ancient DNA path. Simplest to implement, no BAM handling. Two drawbacks: ancient DNA loses the short fragments, and ALE and DeepMAsED end up evaluating a trimmed assembly. It also shifts the coverage values, since reads that used to map to short contigs get reassigned or go unmapped. 2. Map the full assembly, filter afterwards. Assembly QC and ancient DNA keep everything, and the depths stay exactly as they are today. The cost is that we have to filter the BAMs too, one filter per assembly plus one per BAM. Filtering only the FASTA and leaving the BAMs untouched is not an option, I tested it: COMEBin dies with a

KeyError

in

gen_cov.py

and MetaBinner with an

IndexError

in

component_binning.py

, both because a reference present in the BAM is missing from the FASTA. MetaBAT2, MaxBin2, CONCOCT and SemiBin2 tolerate it and return identical bins either way. Thoughts <!subteam^S08MYEKMNTX>?

Has anyone tried github.com/huangnengCSU/compleasm? On paper seems quite an improvement over BUSCO.

Hackathon thread / brainstorm! Hi everyone! I'm a PhD student at Université de Sherbrooke 🇨🇦 , working on multiple sequencing projects in microbial ecology, and a new contributor to nextflow/nf-core. I'll be attending the nf-core Barcelona hackathon catalunya in October. I'm curious to know if anyone else in /mag will be there? The main reason for this thread: I'd like to gather ideas for a project submission, leveraging the hackathon to push improvements for /mag. Submitting a project ahead of the hackathon will help recruit participants to work on those ideas. I identified and discussed a few roadblocks that became issues when processing large amounts of samples, but I have no idea of the scope of work we can tackle over two days (which will certainly depend on how many people hop on the mag train). For that reason I'd love to provide a list of potential projects/tasks/etc. that we collectively think could improve mag, and that might motivate more people to join the effort! So far, we discussed : • QUAST alternative: quast is overkill for what it's required for, quickly clutters work/ (~250 files per bin) causing inode quotas busts for larger projects (on most Canadian supercomputers at least; see https://nfcore.slack.com/archives/CE9MS66BS/p1786628321715299). Some suggested seqkit stats, seqfu, or creating one from scratch. If it can run multiple files in a single job, all the better. • Bin entry point : add an additional entry point for the pipeline, by providing bins directly; would be useful for larger projects, allowing work/ cleanup post-binning, and increasing the overall modularity of /mag . That's it for now. Looking forward to hear your ideas and to know who will be there on October 15-16th !

Hey, all! I'm trying to help a client plan a shotgun sequencing run and wanted to consult with some experts on a few bits. In addition to species identification (which I understand requires less read depth) the client also wants to do MAG assembly. Here are my questions: Long vs Short Reads: I noticed both short reads and long reads are acceptable inputs for the mag pipeline. For this project, we can do either 2x300 bp short reads or ONT / PacBio long reads. From your experience, do you suggest one over the other? Coverage: Also, I've seen quite the range of recommendations regarding coverage per sample. With our large short read flowcell we could get 13 Gb of data per sample whereas with the long read we would be getting closer to 5 Gb per sample. Do you guys have suggestions on the coverage we should shoot for? These are rat intestinal microbiome samples. Thanks for any and all advice you can provide that might help us plan a better sequencing run!

<!channel> As @James Fellows Yates teased some time ago, here is the second nf-core/mag paper in Bioinformatics! Community-driven updates for comprehensive long-read metagenomics and enhanced binning in nf-core/mag v5 academic.oup.com/bioinformatics/…/8770536 This one focus on the enhancements over the version covered by the first publication of nf-core/mag, including long-read only assembly, additional binners and several other extra things added up to v5.5.0. We also did run mag on a few samples to estimate resource usage, so give that a look if you're interested. Thanks to everyone contributing to both building the manuscript and also the helping with the code, specially @Adam Rosenbaum for the lr-only implementation, @Daniel Straub for getting the new bunch of binners, @Jim Downie for always helping with reviews and of course @James Fellows Yates, for coordinating the efforts to publish this, and also because this article covers the whole time when he was leading the development!

The

checkm_download_url

param has a default pointing to Zenodo, which didn't resolve when I ran the pipeline. I wasn't going to run CheckM, so this wasn't needed but still stopped the pipeline from running because of an

"exists: true"

in the schema. This fixes that, with a test case: github.com/nf-core/mag/pull/1109

PR for dereplication with Galah: github.com/nf-core/mag/pull/1110 I've tried to keep it lean and avoided at least one param (`--dereplicate-across-study`; I think we had alternative name that would take a value) that we discussed. I'm happy that add that to this but prefer to defer until we have more experience. There are likely other implementation details we could discuss and for those my feeling is the same: keep as is if it's not a show-stopper to be able to collect experience, issues etc. I have run this on the default test plus a project of my own. In the latter, the number of genomes decrease around 10-fold to 500 at 95% ANI (47 samples assembled individually). Implemented with Claude under my oversight. No extra code review (I will run one now).

Hello, I had a question about the

--ancient_dna

parameter and pydamage (I am using mag version 5.5.0) . I was wondering if the pydamage filters our contigs that do not pass ancient DNA authentication before assembling bins? I am looking out the output information of my assembled bins and it looks like some of the bins contain no dna degradation patterns. Here is my code: nf-params.json

{
    "input": "\/projects\/standard\/mnievesc\/zolik006\/01_projects\/01_PIALQ\/02_analysis\/03_Metagenomics\/01_analysi$
    "outdir": "\/projects\/standard\/mnievesc\/zolik006\/01_projects\/01_PIALQ\/02_analysis\/03_Metagenomics\/01_analys$
    "megahit_fix_cpu_1": true,
    "spadeshybrid_fix_cpus": 32,
    "clip_tool": "adapterremoval",
    "save_clipped_reads": true,
    "reads_minlength": 30,
    "host_fasta": "\/projects\/standard\/mnievesc\/shared\/ref_seqs\/human_g1k_v37.fasta",
    "host_fasta_bowtie2index": "\/projects\/standard\/mnievesc\/shared\/ref_seqs",
    "save_hostremoved_reads": true,
    "gtdbtk_single_job": true,
    "gtdb_db": "\/projects\/standard\/mnievesc\/zolik006\/02_refseqs\/10_gtdbtk\/gtdbtk_r232_data.tar.gz",
    "prokka_with_compliance": true,
    "prokka_compliance_centre": "UMN",
    "binning_map_mode": "own",
    "min_contig_size": 500,
    "save_assembly_mapped_reads": true,
    "exclude_unbins_from_postbinning": true,
    "run_checkm": true,
    "checkm_db": "\/projects\/standard\/mnievesc\/zolik006\/02_refseqs\/09_metaWARP\/CheckM_DB",
    "refine_bins_dastool": true,
    "refine_bins_dastool_threshold": 0.299999999999999988897769753748434595763683319091796875,
    "postbinning_input": "refined_bins_only",
    "run_gunc": true,
    "gunc_db": "\/projects\/standard\/mnievesc\/zolik006\/02_refseqs\/11_gunc\/gunc_db_progenomes2.1.dmnd",
    "ancient_dna": true
}

I ran the same code without the

--ancient_dna

and it seems to make the same contigs and bins as when I had the parameter on. So does this parameter affect downstream results, or just produce information that the user can then go through and determine if bins contain authentic aDNA?

Hello, I had another question about how the pipeline filters contigs. Is there a minimum depth of contigs that the mag pipeline is looking for before assembling bins? Most of the mins assembles are shown to be about 10x (but went as low as 9), I was wondering if there was a parameter that I could use to reduce that? Maybe to 2x?

Hi everyone! I need some help with an

nf-core/mag

run. I was running the pipeline on a set of samples, but 4 samples were repeatedly failing during the SPAdes step due to memory limitations. I therefore removed those 4 samples from my samplesheet and tried to restart the pipeline. However, instead of continuing from the completed steps using the existing cache, Nextflow appears to be starting the pipeline from the beginning. I really want to avoid rerunning all the completed steps, as the pipeline has already been running for a long time. What would be the correct way to remove these 4 problematic samples and resume the existing run with only the remaining samples, while retaining the cached results from the previous run? I am using

-resume

, and I have not deleted the

work/

directory. Any advice on the correct approach would be greatly appreciated. Thank you!

A new binner that claims better performance than ComeBin while also being much faster: nature.com/articles/s41564-026-02472-7

❌ mag/test_full failed 😿 Run Name distraught_thompson Duration 1.6s Status ❌ Failed Error Message

Script compilation error
- file : /.nextflow/assets/.repos/nf-core/mag/clones/170131e6dc670f20a0bd79a78ccfd3f8dbe48547/main.nf
- cause: The LHS of an assignment should be a variable or a field accessing expression @ line 61, column 28.
       longreads_min_quality: Integer?
                              ^

1 error

Command Line

nextflow run '<https://github.com/nf-core/mag>' -name distraught_thompson -params-file '<https://api.cloud.seqera.io/ephemeral/cJ0JD9EADsbX_DkIRTZ-CA.json>' -with-tower -r dev -profile test_full

Oct 7, 2026 at 9:49 AM

❌ mag/test_full failed 😿 Run Name silly_descartes Duration 1.7s Status ❌ Failed Error Message

Script compilation error
- file : /.nextflow/assets/.repos/nf-core/mag/clones/7fbb2153e338225380fe64d22148de7a044de899/main.nf
- cause: The LHS of an assignment should be a variable or a field accessing expression @ line 61, column 28.
       longreads_min_quality: Integer?
                              ^

1 error

Command Line

nextflow run '<https://github.com/nf-core/mag>' -name silly_descartes -params-file '<https://api.cloud.seqera.io/ephemeral/kCmhT2Qvldu0iTuwrXNwew.json>' -with-tower -r dev -profile test_full

Oct 8, 2026 at 8:09 AM

Do we have mag-inclined folks at the hackathon this week? I suggested working on some improvements to our already great pipeline. Curious to see if there is interest already? https://nf-co.re/hackathon-projects/hackathon-barcelona-2026/mag-improvements

depictio Depictio dashboard for mag: your feedback wanted Hi all! 👋 I am building Depictio, an open-source platform from EMBL to explore pipeline results without writing a Shiny or Streamlit app each time. A pipeline gets a dashboard template: a short definition that turns the outputs of any run into an interactive dashboard. I am now sharing it more widely, so I would like the view of mag users and developers. ✨ What it does • MultiQC reports (aggregated across runs) next to result tables, images, phylogenetic trees, maps... with linked filters • Reusable: point the template at any new run of the same pipeline and you get the same dashboard • Fully editable: modify any dashboard from the GUI or in YAML, so the template is a starting point, not a constraint • Versioned, with comments and annotations on any component • Runs as a Python package, with Docker Compose or on Kubernetes, and can serve as a companion web app for a publication 🧪 For mag: assembly contiguity, contig coverage per sample, bin completeness and contamination, bin taxonomy, annotation and a per-bin detail view, next to the MultiQC report. 👀 What I need: I have built 25 templates and want people who really use these pipelines to review and criticise them. What is useful? What is misleading? 👉 Dashboard: mag dashboard (no account needed) 💬 Comment on the dashboard (Google sign-in) or in the GitHub discussion dedicated to mag. ⭐ A star on the repo is welcome if you find it useful. Thanks! 🙏 🎤 I was speaking about Depictio at the Nextflow Summit (session page), and the recording will be available in the coming weeks.

Are there plans for a 5.6 release? I have modified magmap to take input directly from #CE9MS66BS with a param that just points to the output directory of mag. It currently depends on dereplication so I can't ship that until mag is released. I think I will post a "post-binning entry" PR following the github.com/nf-core/mag/issues/623 discussion today.

Fairy thread (issue #796 @Jim Downie) Trying to figure out exactly what the use case is. Seems Fairy would be useful only if neither CONCOCT, ComeBin and Semibin are being used, in which case you actually need the BAM files (this is from a short discussion with Claude, haven't validated all the actual dependencies by hand). Fairy would come in handy when not using any of those binners, only using Maxbin, Metabat, and/or Metabinner. So fairy could be useful for a "fast binning mode" that includes (at most) only these three. Is this somewmhere we want to go? Or am I reading all of this wrong?

Yes, I agree the use-case is harder to see now - previously all the binners took depths files, but we have since added binners that will only accept BAM

Bakta PR incoming for github.com/nf-core/mag/issues/346. Few questions to help me wrap it up: • in the

conf/containers_*.config

should I have something equivalent to

process { withName: 'PROKKA' { conda = 'modules/nf-core/prokka/.conda-lock/linux_amd64-bd-f21b98bcef4c3579_1.txt' } }

for bakta? • Not sure what the multiqc config numbers mean for column plcaement. Is it just like a convention for column order by increasing number? • prokka has a -fast mode that essentially skips CDS/product searching. An equivalent param in Bakta is --skip-cds but I'm not sure if for the pipeline bakta_fast_mode is appropriate, let me know what you think.

I helped with 1/2, point 3 we will need a discussion about as there are A LOT of skips parameters

Personally I would keep it simple and just leave ext.args for customization for now

There it is! github.com/nf-core/mag/pull/1129

HI all, I am using nf-core/mag 5.4.1 The output of checkm2 and busco is missing for the all the bins from several samples (not the same for busco and checkm2). Does anyone have an idea of what happened? The workflow completed without error.