I'm trying to implement modern Nextflow options in #C06R4FC4C1E and I've run into an issue with module

resources

directories. My local scripts use a shared

utils.py

for some common networking logic. Now that I'm moving to the new structure, it must remain available to all the scripts. I tried to symlink the utils file in all relevant places, but Nextflow is very unhappy with this. Is there a clean solution to this, other than "copy and pray you never have to change it"? (For that matter, I am planning to change it later)

The

bin

directory in the root directory of the pipeline should always be in the

PATH

(and there is also a

lib

). The problem with that is that you cannot "package" some part of that with a module (and ideally you keep all of the things a module needs together), but if you want a single copy for all the modules that need it, that's one way to do it. Other than that, you could have hardlinks (rather than symlinks), but the problem with those is that depending on how things are being copied (or even edited), they may not be preserved (and AFAIK

git

is not aware of them, either), so that would mostly be a solution for your local copy.

I found a solution using Seqera AI, a bit hacky, but it seems to work fine. The idea is injecting the pipeline-level

bin/

into Python

sys.path

at runtime. You can see the implementation at https://github.com/nf-core/reportho/pull/116, hopefully no heresy has been committed.

I'm not a Python person -- instead of telling it to use a very specifc path which the

which

command found in

PATH

, is there no way to directly tell it to use the script wherever it can be found in the

PATH

? Or (rather than the detour via

which

, which may fail, because sometimes it gets redefined, like on the system I'm using), could you just use

$projectDir/bin

(if you need to use a specific path)?

That would require porting from a module binary to a template, I would have to evaluate how complicated that would be (since currently some of the local modules combine Python with Bash). I will have a look into this.

Yeah, I think templates are not really encouraged. Maybe you can pass the path as a parameter?