When running tests on fast multi-cpu server I noticed that some steps are still using single cpu, for instance, CELLTYPES_SINGLER, and it takes several minutes on each sample. Added details in the issue.

Most steps are actually inherit single-threadedness from their upstream implementation, but in cases like singleR it seems like nobody noticed yet that the connection between the available and utilized cores has not been made

I think that's why

nextflow run nf-core/scdownstream -profile test,docker --outdir results -r dev

took almost one hour on 10 CPU / 20 Gb RAM. Most of the time I saw only 1 CPU utilized... I'm now rerunning it -with-report, so we can see what can be optimized (if possible of course, I remember many bioinf tools which can't be)

There should be a difference between the per-process multithreading and the nextflow-level parallelization (e.g. starting multiple tasks at the same time). If you only see a single CPU utilized, this most likely means that nextflow does not properly parallelize because it thinks it does not have enough resources for parallelization. This is most likely because of the

resourceLimits

directive that is part of the

test

profile. This limitation is there because it needs to run on the github action runners, which have quite limited resources You should probably override this so that nextflow uses more of the actually available resources

I just think that it would be great if the minimal test suite can be sped up... My teammates running it on laptops reported 2+ hrs, which is a bit frustrating

Actually, I have to revise some of my earlier statements: The

resourceLimits

is not 'what nextflow is allowed to utilize from your computer' but rather the maximum resources that a single task can request. This needs to be lower than your existing resources, because otherwise it can lead to failures, especially related to the dynamic resource scaling on retry. So increasing this won't increase parallelization On the other hand, now that I have been thinking about what you reported a bit more, I remember I have seen the pipeline stuck on singleR for a few minutes every time the tests run, so implementing the multithreading properly will probably be beneficial

I can rent a faster server, but what's the point if most of the time it will be almost idle (most of the time not much of the RAM is used either). I'll look into reports in the morning. Thank you for your detailed response, @Nico Trummer !

For most use cases (e.g. the addition of the PCA as integration method) you won't need to run the entire

test

profile. The pipeline comes with tests for each module and each subworkflow. so when you add the PCA as integration method you should be fine by just running the tests of the integration subworkflow and the PCA module, and once it works, open a PR. The CI will initially fail because the global test snapshot a list of expected output files, and changes like adding a new integration method will trigger a mismatch. Updating the caches requires running the test profile ~5 times, as we are testing many different configuration combinations of the pipeline to ensure all relevant use cases are working properly and reproducibly. I can run this on my HPC cluster which has in total around 200 times the compute capacity of your rented server, and I don't need to pay for it. So if you guys open a clean PR and tested that the module tests and the subworkflow tests pass, I am already happy. The rest is then relatively easy but takes long. I usually do these things in the background while working on other things