<!channel> Am I in the right meeting? Right time?

@Daniel Lundin we are waiting for you in the meeting

you may have to re-enter the pwd manually 154239

eyebold.dev.lhst.eu the cleaned bold database Giorgos mentions

biorxiv.org/content/10.64898/… MSA difficulty prediction

Thanks everyone who joined for the meeting Tuesday! I have been thinking some more about the fate of "my" part, the pipeline and I believe I'd like to push on an propose this to nf-core as an official pipeline but a bit modified compared to today. @auguste.gardette, you have clearly improved the main bottleneck in a better way than I was into, and that also will better benefit from any improvements in the initialization of EPA-NG. To include your script in the pipeline it would be practical with a Conda recipe. I got a couple of issues from Auguste, if anyone else has found bugs or want features, please go ahead. I think it would be good to give the pipeline a different name. When I started, my ambition was to replicate Sativa, now it's more to provide a complete taxonomic marker test pipeline, maybe: nf-core/taxmarker? Any other ideas? (For a later discussion: I still have the feeling that using Nextflow's parallelization could be beneficial. Not by letting each LOO step run in its own EPA-NG session, but maybe batches of some number of placements. That could provide a way to scale from the tens of thousands to hundreds of thousands or millions as I think I mentioned. For this, the creation of LOO tasks would have to be separated from their placement and evaluation. Would it be possible -- and not too much trouble -- to provide partial exit and entry points to your script, @auguste.gardette? I.e. with a param say that the caller wants the LOO tasks, wants to execute the EPA-NG placement, the evaluation etc.? What I'm after is keeping it standalone, while still supporting discrete steps in the pipeline. If that's too much work, unwieldy or something or just something you prefer to defer to later, I can easily build the current pipeline with a single Sativa step.)

All good, I am just not sure what you mean by parallelization as batches of placements, is this batches of LOO tasks grouped together or do you want to further parallelize each LOO task by conducting simulatenous placements into different branches, the latter is implemented in EPA-NG anyway, so I would opt for parallelizing over LOOs, but assign more than one core to each LOO task such that it can profit from EPA-NG's internal parallelization that parallelizes over placements

Parallelize over batches (say 10,000) LOO sets each sent to one instance of EPA-NG to benefit from the improvements we discussed Tuesday. I hope I'm clear there. 🙂 I'd aim for a self-contained Sativa first and do this when we see benefits. However, I also realized that the building of the phylogeny from the taxonomy (first two steps) could be factored out (as they are in my current implementation) since they clearly have different resource requirements than the later stages. Also for later discussion.

<!channel>: nf-core/taxmarker proposed to the nf-core community: https://nfcore.slack.com/archives/CE6SDEDAA/p1787823600443959 Already with one approval from a core member.

Okay, I got it, but if memory requirements become too large you may still want to opt for a hybrid approach, parallelize over LOO sets at a coarse scale and internally over placements within each LOO task, like this large memory requirements can better be handled.

Exactly what I was after. Do whatever parallelization is possible in Sativa itself, but use Nextflow for coarse-grained paralllelization by running each individual step as one Nextflow process, branching out in large batches over the steps that could be made to scale beyond what Auguste mentioned was the limit today. This would require splitting up steps from Sativa either through params or different scripts. Basically relieve Sativa from the workflow orchestration.

@Daniel Lundin : I've published a release of the EPA-ng version of SATIVA, tagged

v0.9.3.3

, and opened a bioconda PR for it (package

sativa-epang

). SATIVA can now be called step by step: (i) build the reference, (ii) prepare the folds, (iii) place, (iv) score. Each step can be its own Nextflow process now. Each fold directory holds everything needed to place it (the tree, the alignment, the queries, the model) along with the EPA-ng command to run in it. So the placement step needs neither the full reference, nor the taxonomy, nor even SATIVA installed. Run with no options, SATIVA still does everything on its own, as before. The code is here: github.com/Aaramis/sativa-epang :

CHANGES-epa-ng.md

documents the steps, the fold layout and the manifest.

hey everyone, im aaron, im a bioinformatician down in chile, @auguste.gardette told me about this rework and Ill dive deeper soon and see if I can contribute in some way :))

Hi. I've been away at a conference and vacation for the last 3 weeks. Exciting to see all the ongoing efforts to enhance SATIVA. I was (am?) writing an alternative to SATIVA, but this was primarily to bridge the high computational requirements of SATIVA. My primary use case is doing a QC of all 16S rRNA sequences identified in Genome Taxonomy Database (GTDB) genomes. For reference, this is ~1 million 16S sequences and ~100,000 sequences after clustering at 99% identity. @auguste.gardette, will the improvements you made support QC'ing a set of 1 million 16S sequences? Or the ~100,000 sequences after dereplication?

@Daniel Lundin Quick update: the Bioconda PR has just been merged, and

sativa-epang

is now available on Bioconda 🎉 anaconda.org/bioconda/sativa-epang

has renamed the channel from ‘sativa’ to ‘taxmarker’

Hi @Daniel Lundin - I've just approved your pipeline proposal: github.com/nf-core/proposals/issues/165

I renamed this channel to

#taxmarker

instead of creating a new one, I hope that's ok! Shout if you'd prefer not to.

To me it looks like the next steps are: • Rename github.com/erikrikarddaniel/nf-core-sativa/tree/dev to

nf-core/taxmarker

• Put in a request at #C09H6NYHR9T to move the repo to the nf-core org • Make

dev

the default branch, up until time of first release, plus other items listed in the docs

I've just made a new branch for you in #C02L5UB4Y9G: github.com/nf-core/test-datasets/tree/taxmarker

If possible, I think it would be good to make a new channel and keep #sativa. The former for the pipeline, the latter for the tool. The pipeline was initially supposed to replace the tool but has become much more and some people already in the channel might want to just discuss the tool.

has renamed the channel from ‘taxmarker’ to ‘sativa’

no worries, done!

ok and I've removed everyone again from core that I added as part of the new-pipeline-channel process 😅

@auguste.gardette: I've (Claude actually) started building nf-core modules for your tool. We're finding some small issues. Here are the two current ones: "Stale internal version string. epac/version.py still has SATIVA_BUILD="v0.9.1" / SATIVA_RELEASE_DATE="2023-08-16" — untouched since the fork's SATIVA v0.9.3 base, even though the bioconda package is at 0.9.3.4. Every run's own startup log (SATIVA_INFO % progname) claims to be "v0.9.1," which is actively misleading for anyone debugging or filing a bug report against a specific release. Combined with there being no --version/-version flag at all, there's no way for a caller (us included) to ask the installed tool what version it actually is — I had to reverse-engineer it from the conda package's installed path (share/sativa-epang-<version>-<build>/, baked into the sativa-epang wrapper script at build time) rather than anything the tool reports itself. Worth suggesting he bump SATIVA_BUILD on each release tag and/or add a trivial version flag. 2. Silent GTR+G fallback if -refmodel is omitted with -reftree. The help text says -refmodel's default is GTR+G — but nothing stops you from passing -reftree without -refmodel, so a caller who forgets the model flag gets scored under a silently-substituted, possibly-wrong substitution model with no warning, rather than an error. Since our module always passes both together, this doesn't bite us — but it's a footgun for any other caller of the staged CLI. Worth either erroring when one is given without the other, or at least logging a visible warning." I suppose we should post that as issues on your repo, correct?

And one more tiny comment: "the log's step label still says "Calling RAxML-EPA to obtain branch labels" even on this path, but the actual tiny.raxml.log timestamp is untouched from our own earlier RAxML-NG run — meaning EPA-ng ran, not RAxML, and the log message is just a stale leftover string (same "renamed the code but not the print statements" pattern as the stale version number). Cosmetic, but worth a one-line fix on his end too since it actively misleads about which engine ran."

It looks like I can't post issues on your repo -- I guess it's because it's a fork. That brings up the question whether this will actually be considered a fork in the GitHub repo sense or the wider sense of a new repo/project taking over where the old left off?

We -- me and Claude -- think it's time for 0.9.4 or 0.10.0 -- including a new Bioconda recipe -- what do you think @auguste.gardette? (The bioconda automated PR is apparently not for the latest commit.) We would then base the nf-core modules and subworkflow on that.

For those of you not in the #C0BVDK00QKY channel, I just posted a message regarding pre-alignment clustering in the pipeline. Any comments welcome: https://nfcore.slack.com/archives/C0BVDK00QKY/p1789035585327989