Hi <!channel> sorry for the ping, but I really do need an answer to the above question. I assumed that I'm correct and that each run of the pipeline assumes all samples are from the same species, because of the step where it pools all the runs. I started running them separately while waiting for an answer, but I have enough species (57) that I am running into the file number limit on my scratch because of how many intermediate files are generated (somehow even 10 runs at a time generates 1M files??). If I'm wrong and could be running this as one big pipeline, that would be great to know. Thanks, Serena

Any idea @Avani Bhojwani?

As you say, it definitely pools all reads from files in the input sample sheet into a single file before proceeding with the pipeline, so by extension I don't see any way for it to consider the species separately if input together.

sorry for the late response. Yes - reads from all samples are pooled together before commencing assembly, so you would have to run the pipeline multiple times if you would like a separate transcriptome for each sample

I definitely second @Isaac W’s request to be able to run multiple species in one run of the pipeline! I do have

cleanup = true

in my config, but as I read the main nf-core docs, it looks like it only gets rid of unnecessary temp files once a run of the pipeline has completed successfully, which doesn't really solve the issue of having these exploding trinity tempfiles. I set up a little script to track how many files are being produced, and

cleanup

does get rid of everything besides the `.command.sh`/`.command.log` files at the end of a run, which leaves you ~25 files per run; all well and good, but doesn't let you run more jobs at once, especially since running as a job array means none of the runs complete successfully when the file number cap is hit. I've been using the slurm

#SBATCH --array=1-57%1

function, where the number after the

%

is the number of jobs that can run at a time, to try and do something similar to what @Isaac W did, except without the nice features of checking scratch usage and stopping subsequent submissions. I kept trying with successively smaller and smaller numbers, but once I hit doing only 10 at a time and was still generating more than a million files, I quit and asked our sysadmins for a temporary increase for the file number quota 😂 they haven't approved it yet & I'm not sure they will, so I'm just running jobs one at a time while I wait, and they are moving a little more quickly than I expected, which is nice. That being said, 57 transcriptomes is not that many, and having the pipeline create such a prohibitive number of temporary files is a huge barrier to being able to use it. Not sure if putting them all in one pipeline run would actually solve that, if it's trinity making separate tempfiles for each assembly, but it would at least make it easier to use!

@Avani Bhojwani thanks for making this GitHub issue to track this suggestion! https://github.com/nf-core/denovotranscript/issues/48