Hi everyone! I have a question about processing many (~1000) samples with taxprofiler and a large Kraken2 database (300Gb). memory mapping did not work for me, and running each sample as an individual process with a 300Gb Kraken2 database is painfully slow. Is there any way to submit multiple samples to the same process to ensure a precached database? Any other solution?

You'd have to make bigger changes to the workflows. We have done what you describe for krakenuniq, so you can take inspiration from that.

Yeah, we would need a new special module that uses this parameter:

--use-daemon          Spawn a background process that keeps any loaded indexes in memory. Subsequent invokations of classify with
                        this option will skip the index loading process and immediately start classifying reads. If a new index is
                        specified that index will also be persisted. Use k2 clean --stop-daemon to stop the background process.

Hmmmm alright. Thanks for the useful responses, Ill have a think whether Ill try and push through all the samples or if it makes sense to implement a solution

Hmm, do you have some control over the nodes or are you completely at the mercy of the scheduler?

Control in what sense? I cannot circumvent the scheduler if thats what you mean

Control in the sense, can you determine the pool of nodes for scheduling and can you perform operations there interactively? Like setting up an in-memory filesystem and loading the database there?

I can schedule an interactive node. I am not sure how I would tell nextflow to use an existing allocation?

Right... It would be a little difficult, but you could schedule an interactive node of sufficient size (I don't know how long you can hold it for). You would then create the right setup for kraken2's memory mapping there. Start nextflow on that node and configure it to run the kraken2 step on the local executor.