Question: quite a few modules set a temporary directory option to './tmp' or somesuch so that a task's scratch files are written relative to the task working directory, with no option to override:

rsem-calculate-expression \
    --num-threads $task.cpus \
    --temporary-folder ./tmp/ \
    $alignment_mode \
    $strandedness \
    $PAIRED_END_FLAG \
    $args \
    $reads \
    $INDEX \
    $prefix

github.com/nf-core/modules/blob/…/main.nf#… Can we encourage a different idiom that allows an override for folks who are confident in doing so:

def args = task.ext.args   ?: ''
prefix   = task.ext.prefix ?: "${meta.id}"

if (!args.contains('--temporary-folder')) {
    args += ' --temporary-folder ./tmp/'
}

Then folks can e.g. supply the system TMPDIR when they're confident in it, and folks using e.g Fusion that want a scratch dir that doesn't stream back to S3 can override as they want. Does that sound reasonable? Example: github.com/nf-core/modules/pull/13043

I agree that modules shouldn’t hardcode using the working directory without any option to override. But you’re saying that for a tool that uses TMPDIR by default, you want to override that with the working dir at the module level, leaving an option for users to fall back to TMPDIR by overriding

ext.args

? That still sounds convoluted. Why not just make the module not do anything, and add

--temporary-folder ./tmp

to

ext.args

in your pipeline / institute config as needed. On HPC where the scratch dir is a network filesystem, users will pay the price of using the working directory immediately. And how should they know that the module has been diverted away from the default TMPDIR ?

Right now I'm just keen to be able to avoid

--temporary-folder ./tmp

(the hard-coding thing), and so far I've preserved default behaviour. Are you suggesting it would be better to just strip all default temp dir specs and leave it to users to do via ext.args without module-level defaults?

Yes. Because I’ve faced many tools that do a lot of temp I/O and it just thrashes the network filesystem. Though to be fair, I’ve also seen some tools use so much temp space they fill

/tmp

and in this case we have to use the working dir (network filesystem) and it’s slow …

I'd be on board with that, it's just a potentially more disruptive change. Though maybe we just make the recommendation going forwards.

Why not just make the module not do anything,

I think a lot of these hard coded things are there because people have experienced problems and not doing anything was not optimal. I do agree though with the rest of your concerns. For the time being, I would say make the temporary folder an optional input to make it clear that this tool has issues with tmp space, and stick with whatever was implemented by the module author, but now it's overridable.

sounds like a perfect use for a

task.ext.tempdir

argument...

Thanks, yep, that's what I've done

Yeah, I see. The trickier ones are all the tools that use

TMPDIR

or similar

Right, I feel you. And related to this, Nextflow doesn’t mount

$TMPDIR

by default.: github.com/nextflow-io/nextflow/issues/4587#…