Thanks everyone who joined for the meeting Tuesday! I have been thinking some more about the fate of "my" part, the pipeline and I believe I'd like to push on an propose this to nf-core as an official pipeline but a bit modified compared to today. @auguste.gardette, you have clearly improved the main bottleneck in a better way than I was into, and that also will better benefit from any improvements in the initialization of EPA-NG. To include your script in the pipeline it would be practical with a Conda recipe.
I got a couple of issues from Auguste, if anyone else has found bugs or want features, please go ahead.
I think it would be good to give the pipeline a different name. When I started, my ambition was to replicate Sativa, now it's more to provide a complete taxonomic marker test pipeline, maybe: nf-core/taxmarker? Any other ideas?
(For a later discussion: I still have the feeling that using Nextflow's parallelization could be beneficial. Not by letting each LOO step run in its own EPA-NG session, but maybe batches of some number of placements. That could provide a way to scale from the tens of thousands to hundreds of thousands or millions as I think I mentioned. For this, the creation of LOO tasks would have to be separated from their placement and evaluation. Would it be possible -- and not too much trouble -- to provide partial exit and entry points to your script, @auguste.gardette? I.e. with a param say that the caller wants the LOO tasks, wants to execute the EPA-NG placement, the evaluation etc.? What I'm after is keeping it standalone, while still supporting discrete steps in the pipeline. If that's too much work, unwieldy or something or just something you prefer to defer to later, I can easily build the current pipeline with a single Sativa step.)