Another question on module practices. If a tool outputs a generically named file (e.g.

stats.txt

) instead of one with a customisable prefix (e.g.

{prefix}.stats.txt

) is it acceptable to have the files renamed within the module using

mv

? I have found this situation is quite frustrating when a file is used downstream and there are name clashes.

Yes, that is allowed!

when a file is used downstream and there are name clashes.

Remember also that you can stage files into a process with wildcards to avoid this

Yeah but I think to reduce having to mess with modifying inputs (which can have unintended side effects) we prefer to 'standardise' outputs

The issue is that staging is also a module level piece of code. So you might have to modify any number of modules which did not expect to have to deal with name clashes. A few in

nf-core

do use e.g.

stageAs: 'input/*'

but I find that it’s rare

Yes, definitely feel free to rename outputs using

mv

@Adam Talbot can you remind me (I think it was you) how you suggest people should rename stuff... I vaugely remember someone saying such operations are bad for filesystems or something something

(I would also put the tool name into the filename as well as just

stats

)

For me the additional prefix is up to the user

So how would do people do that?

ext.args.prefix

in modules.confg

But that controls the main output file

It should apply to all?

But the point here is it doesn't

You can't also control the secondary files that way

(Unless I misunderstand?)

We don't really make a distinction between primary/secondary files normally. Typically tools/modules give everything the prefix or nothing

Sure, it all needs the prefix. If you are calling the output from every tool

${prefix}.stats.txt

you end up with similar issues downstream, but we don't worry about that for e.g.

${prefix}.bam

so maybe it doesn't matter anyway. And multiqc sorts it out by just staging everything in separate files anyway

But I mean a developer might not want

stats

on everything, they may want to distinguish with another term

Prefix should be unique between process invocations ideally

But I mean a developer might not want

stats

on everything, they may want to distinguish with another term

But we can't do that without also changing the name of the actual output file

> can you remind me (I think it was you) how you suggest people should rename stuff... I vaugely remember someone saying such operations are bad for filesystems or something something Meh, no hard rules. If you need to rename a file do it with

mv

. Rules: 1. Accept that hitting the filesystem is sloowwwwww so you should try to avoid it. If your tool writes to

-o ${prefix}

do it! Do you really need to rename the files? Really really really? Nextflow handles the file stuff for you so you can normally avoid it, e.g. pass around the tuple

meta, output.txt

and rename on publish. 2. Don't assume you have a full, happy complete filesystem and can do any of the weird tricks. For example don't do

ln

to rename files. 3. Remember not everyone is using the same filesystem as you, so avoid doing something because it's super fast for you. For example,

mv

is often a

cp && rm

on my many different filesystems. To summarise: keep it as simple as possible and no simpler.

Hmm in

nf-core/modules

modules there are: • 30 modules using

ln -s

• 135 modules using

mv

• 47 modules using

cp

How bad are these, how often do they break?

Well they're obviously working enough for now? I would look at the motivations for using them and see if they're necessary or the authors were taking a shortcut.

cp

and

mv

are inefficient but work, but

ln -s

doesn't work in a number of situations.

mv

I think is used often to either rename or move files to subdir for capture. Occasionally it seems to be used as part of the process for e.g. renaming tmp files.

ln -s

I think is mostly used to rename existing input files. Here is one justification:

// Added soft-links to original fastqs for consistent naming in MultiQC

It is also used in a some places to put input files in the correct path alongside other index files for a tool (e.g.

biscuit align

)

cp

seems to be used mostly as an alternative to

mv

for renaming (this feels like it is probably bad)

Sounds like they could all be replaced with

mv

I would guess that

mv

is not good for renaming inputs since that would • break resume • often target symlinks • mess with the state of files possibly coming from outside the pipeline

cp

inputs,

mv

outputs?

@Adam Talbot wannnbba make a PR with a proposal for updating the module specsm

I think you have to be a touch careful with symlinked directories and

mv

. I'm still unclear whether moving a staged input symlink would break resume though (I thought it would, but maybe you're right and nothing happens)