This is more of a nextflow question, so I am not sure this is the right place to ask - feel free to tell me of if I am stepping out of line 🙂 I created an (internal) pipeline to do genome annotation following nf-core guidance and using sub-workflows, and all is good when I start with about 30 input samples. However when I run the full list of species, > 5000, from a samplesheet, nextflow takes a long time to even start submitting jobs. Is this normal? Are there ways around it? I know some of you are submitting very large samples number samples so maybe I am doing something wrong. Details: • it's my first pipeline in nextflow and I used AI to help me. • jobs are being submitted on a

GCP

VM instance as GCP batch jobs. Cheers!

Here's a similar thread on the Nextflow slack but we didn't come to a satisfactory solution https://nextflow.slack.com/archives/C02T98A23U7/p1752224037656359 File system operations seem to be the major hinderance, but we also didn't hear from the people regularly processing millions of samples.

Thanks! I don't think i came across that thread when I first searched. I'll check if of those work, otherwise I will bit the bullet and wait for nextflow to stage the jobs. I won't be doing 5000 samples in one go very often

I dunno if it's the same issue exactly, but I encountered something the other day where I had 30,000 microbial reference genomes on a n S3 bucket, and it took 2h for nf-schema to validate the samplesheet and something similar to stage onto the local filesystem... Once that was all done it started going at a more normal soeed