Regarding the linting for the strict syntax when using

templates

, how is that supposed to work? In e.g. https://github.com/nf-core/scdownstream/blob/dev/modules/local/scvitools/scvi/main.nf there are a bunch of variables that are declared and not used in

<http://main.nf|main.nf>

but are used in

scvi.py

through

template(scvi.py)

. Linting shows warnings that the variables are declared but not used. I thought about adding a no-op (e.g.

: "${variable}"

), but then there'd have to be a

script

block (which there isn't, since it's using templates), plus it feels ugly.

How about migrating to module binaries instead of templates? That way you can call your python script in a

script

tag

Do we have an nf-core policy on that? I think we have recommended templates in the past to avoid dependencies on Wave and stuff, though I forget the specifics now.

Yep, module binaries have a run time Wave dependency I think we should avoid for community modules.

Yep, module binaries have a run time Wave dependency I think we should avoid for community modules.

But if we're starting to use Wave / Seqera Containers to generate containers for nf-core modules, then it could be baked in - no Wave dependency at run time? 🤔

I think Nextflow would still try to use Wave to bake the scripts in, even if we hacked around to pre-bake them

Issue opened here: https://github.com/nextflow-io/nextflow/issues/6777

No, I think it will need to stay as a template

Okay, then I'll just wait for a future solution

I think Nextflow would still try to use Wave to bake the scripts in, even if we hacked around to pre-bake them

If there's a limitation that you see with the Wave + module binary approach, you should bring it up in a Nextflow issue so that we can work towards a solution. I think it would be a viable alternative to the module template approach but it's hard to say for sure until someone tries in earnest

I'm not really sure how this would work though 🤔 We do the Wave build in CI using the Wave CLI, so it's separate to Nextflow. We'd have to somehow tell that process that a specific folder should be injected. Also I'm not sure that we allow / want to allow this for the public registry.

Bottom line is that folks expect this to work exactly like the workflow bin, which it doesn't, and I'm a bit worried that some nf-core CI workaround would generate some confusion.

You would use

nextflow inspect

to build the containers exactly as Wave would build them at runtime. I thought Edmund did a POC for this a while ago... The containers would be tied to specific modules though, not just an arbitrary set of conda packages

Yup, and then we'd need to host those built containers somewhere (not Seqera Containers), like quay.io or something

We're really starting to see the limit of templates now that we're migrating to topics for the versions

Completely agreed that module binaries should be able to work for containers without a dependency on Wave! I do understand there would be limitations around e.g. file name clashes, but I don't really understand why module binaries can't operate using the same mechanism as the current staging of binaries in

bin/

, even if it just means dumping all module binaries into a single directory at runtime and staging it

This has been discussed many times. TLDR:

The same mechanism as the current staging of binaries in

bin/

This mechanism is a bit of a hack and Paolo hates it, and would love to remove it from Nextflow if he could. He refuses to make it worse / duplicate it for modules.

I remember. But from a user point of view it's so nice. I put a script somewhere and it magically appears in the container.

I know that. You know that. We all know that.

Any hack that works is a good enough hack for me

But that's why I am not in charge of Nextflow 🤣

This mechanism is a bit of a hack and Paolo hates it, and would love to remove it from Nextflow if he could.

I would accept this if it happened. At least it would force us to take a more declarative approach, rather than dangling a tantalisingly almost-useful feature at us

Just had claude evaluate things, best it came up with is adding scripts as task inputs. 😱

You can do that with the GAWK module...

We could probably have nextflow do it implicitly, I might mock it up just for fun

Interesting idea. With typed processes, it might not even need to be an input. I think you could just use the

stageAs

directive to stage it in:

nextflow.preview.types = true

process HELLO {
  input:
  // ...

  stage:
  stageAs file("${moduleDir}/bin/script.py"), '*'

  script:
  """
  script.py ...
  """
}

Oh, now that I like

Ben, are you telling us we've had a fully portable module bin mechanism all along as soon as types happened?

so that's another reason to go for typed processes

Following along as this discussion unfolds is fun 😉

@Ben Sherman this is the more involved solution Claude and I were working on: https://github.com/nextflow-io/nextflow/pull/6928. It even harmonises workflow bin and module bin in the hope of appeasing 'he who sees all' 🙂

I've used the "include the script as an input" method in the past, for example during development before I can be bothered to make a container containing the script, or for RMarkdown reports where I wanted to be able to change it at some later point if I wanted to.

I do actually think it makes sense to use the same machinery, whether implicitly or explicitly