Hello, is there a general recommendation regarding...
# configs
n
Hello, is there a general recommendation regarding the
array
directive in configs? Besides being rather new, are there any downsides to putting something like
array 100
into a config and just trying to run everything in array jobs?
n
You should only use this for processes that run a large amount of small jobs. Nextflow will always wait for 100 processes to be ready before submitting them to your cluster. This is fine if you for example scatter your samples into 100 parts to analyze at the same time. This can create some large bottlenecks in your pipelines if used incorrectly
so I'd advise to only do that for jobs that have a large quantity of small jobs where the wait time for new jobs is relatively low
n
Ah, so it will try to fill the array before submitting, I see, yeah that makes sense, thanks
n
No I meant that nextflow will wait to submit individuals jobs to the cluster until it notices 100 jobs for example of the same process. ๐Ÿ˜›
n
Not sure if I still get it ๐Ÿ˜„
n
See the documentation for a way better explanation ๐Ÿ˜› https://www.nextflow.io/docs/latest/process.html#array
n
You mean this right "A process using job arrays will collect tasks and submit each batch as a job array when it is ready. Any โ€œleftoverโ€ tasks will be submitted as a partial job array."
Basically, I guess if I have a pipeline with many samples, the first process could just be an array, since there is no waiting required?
n
yeah that's true! unless your pipeline needs to stage the data first ๐Ÿ˜›
p
I'd also start by evaluating if
array
(and job arrays) is beneficial on your cluster at all. A lot of people seem happy with job arrays but when I evaluated it for our resources, I recall it as giving virtually no benefit and possible some drawbacks (but it was quite a while ago so things could have changed, also might have been because of our setup).
this 2
n
I am not sure if it would really give a benefit in my specific case, our cluster is usually not used at full capacity, so scheduling is easy. My understanding was that arrays mainly make it easier on the scheduler, since it may see more jobs than if they all went into queue until the queue reaches 100 jobs? Which drawbacks did you observe?
p
It's been a long time but essentially they were all treated as separate jobs with the difference that it seemed they weren't satisfied by backfill (meaning they were kept around longer, giving more work). We used to have a scheduler sometimes not keeping up and my tests certainly didn't show any improvement. But again, test on your cluster.
Jobs allocated/started quickly is a good measurement of if it's beneficial.
n
Ok, I see. I have tested it a bit but since we are rarely at capacity I do not see any differences, jobs just start ๐Ÿ™‚ At least with SLURM, the main advantage seems to be that if the first member of the array cannot be used for backfilling, the remaining jobs of the array will not be tested, which I guess can ease backfilling on busy queues
t
I have heard from sysadmins that the (SLURM) cluster has to have the resources for the array job (or at least some part of it) available before starting anything, so you would get faster execution with individual jobs. The only reason I use array jobs at all is convenience in writing the scripts when I have to do the same thing for a hundred files. To do that with individual jobs, I'd need a wrapper script that loops over the files and submits them one by one.