A community effort to collect a curated set of analysis pipelines built using Nextflow.

nf-core

If anyone here is using sarek, inside of aws, without using fusion, I'd love to hear from you.

i guess i acutually don't know if it's configured to use fusion or not

<@U09FJ4XBR2R> thanks for the response and investigation

the default setup for Sarek handles S3 pretty decently

obviously there's some overhead copying files

but the caching is good enough it hasn't generally been an issue for us

we had a separate (non nextflow) implementation that used FSx, which was awesome, but was overkill for our actual needs

could I ask what value you're using for
`process.scratch`
and what that actually points to?
• a large EBS volume mounted on a worker? 
• FSx?
• NVMe?
presumably N worker machines execute during a sarek, do they all share the same scratch destination?

we dont use that specific arg. I believe each worker has its own working directory, not shared

It is my understanding that for any cloud platforms `process.scratch` is set to `true` by default.  Don't know what _exactly_ it points to, but I would expect it to be local to the VM, because that's the point of `scratch`.

yes, I had it it set to false, cargo culted in from the use of fusion. setting it back to true after removing fusion seems to have been sufficient to resolve my error. thx to both of you. this is resolved.

I'd reckon that lowering the number of intervals might be more efficient on such a system

The spread and gather strategy we use in sarek is great for an HPC or on the cloud with fusion, but I can imagine the shortcomings