has anyone had the need to run a pipeline in batches? I have storage limits that don't allow me to run most of them on the entire dataset, I have a bash script that manages that, but I was wondering if there was something native to make things cleaner.

Not that I am aware of. There is the

nf-boost

plugin, which removes temporary files in the work directory after each process is done, so this may help with your space issues, but if comes at the cost of destroying the cache, so you cannot use

-resume

(successfully). If your pipeline has a >99% success rate, it may be worthwhile, but not if it is error prone (e.g. because it pushes the resource limits or your system tend to have random issues). If your script is clever enough to check if a run was successful and (only) then cleans up anything you won't need anymore (at least the work dir) before starting the next batch, that may be about as good as it gets.

yes, thanks, I already use nf-boost, then everything is managed by a bash script that pulls the batch data, generates the samplesheet and submits the job. When the job is completed it goes to the next. I was wondering whether there was something already integrated in nextflow to manage this kind of scenario. Other than that, the script does its dirty job, so I guess I'll continue with that solution.