Does anyone have any recommendations or resources they would refer me to for how to best perform benchmarking with nextflow? I have a workflow that I would like to run many times with slightly different parameters because I want to see how one of the processes performs (memory/time usage and a few metrics derived from the output). I am thinking this could be a workflow of workflows, where my higher level workflow determines the parameter values for each specific run and then starts the lower level workflow to perform each run. What I am unsure about is how best to design the upper level workflow that calls each individual run without the runs interfering. I am also not sure what the best way is to systematically obtain and aggregate performance metrics like memory/time usage within a nextflow workflow. Sorry for the long question, I am hoping someone can point me in the right direction. Thank you so much for this community.

No experience with benchmarking, so take this with a grain of salt. • You probably should start each run in a separate directory to avoid interference, and if you keep the outputs in that dir as well, you could later compare that the important results don't change. You also get the files with performance metrics (

execution_report_*.html

and a bunch of text files) in a place where you know which run they belong to. • I don't think you need a special "pipeline" to start the pipeline run -- a shell script with a

for

loop should do. • Comparing actual run times (as opposed to e.g. CPU seconds/minutes/hours) would probably require that you have a server or cluster to yourself, because anything else that is running at the same time may affect resources. Even if you have RAM and CPUs reserved on a cluster, the file I/O speed still depends on the overall load on the file system (be it a shared NFS allocation or the scratch space on a node). ◦ If you do have a system to yourself and actual run time is of interest, you should complete each run before starting the next, so they don't compete with each other. ◦ Even then there would likely be some stochastic factors that influence the run time, e.g. which which processes run on a node in parallel, which may affect the I/O performance. So ideally you average over several runs for each parameter set. • I would make a

-params-file

or config (to use with

-c

), or both (depending on your needs) for each parameter set and then loop over those. You could have associative arrays in

bash

to keep it all organized: ◦ A descriptive string is used as the "key". This could be used in naming the directory. ◦ The "value" in the array(s) would be the parameter or config file. ◦ The loop would be

for paramset in "${!filesarray[@]}"; do ...; done

.

$paramset

is the descriptive name, and

${filesarray[$paramset]}

the parameters/config file to be used in the body of the loop to fire off your pipeline run. • If you don't care about run times (or are in an environment that scales almost indefinitely, like a cloud), you can fire of all runs in parallel. In that case you would use

&

after the

nextflow

command to put it in the background if you are running all of the commands on the same (head?) node. Alternatively, you could submit each

nextflow

command as a separate job. ◦ If you want to run the jobs one after the other and still submit them to a queue, you would need to keep track of them and only start the next one of the previous is complete. • Regarding systematic analysis of performance metrics, have a look at the

pipelin_info/execution_trace_*.txt

files, which have a bunch of metrics for each process in a format that can easily be parsed by a script to calculate aggregate metrics, etc. Hope this is helpful and makes sense.

Thank you so much for your detailed response. This is very helpful!