Hi ! I have two questions and solutions that would...
# modules
a
Hi ! I have two questions and solutions that would make my life easier 😄 1. When we are developing a new module in our #neuroimaging library (nf-neuro) (so far we have more than 50+ modules but still a wip) I feel like we are missing a lot by not providing any information about the
params
used in the module for our users in the
meta.yml
. They have to dig into the
main.nf
. Can we add information about these
params
into the
meta.yml
? 2. I know we can define default value for those parameters directly into the
main.nf
. It would make more sense in my opinion to have default values stored directly into a
nextflow.config
attached to the module. Then, it would make these default value available for both
main.nf
and the
meta.yml
to get these values printed when running "nf-core module info my_awsome_module" for example. I feel like these questions are really specific to the development of a library, I know there are a lot of things done as a pipeline perspective but here it's a little bit different. Thank you in advance 🙂 @Florian Wuennemann, @Anthony Gagnon Follow up on our discussion 🙂
p
Pretty soon, Nextflow is going to start complaining about "global" params. Instead they should be passed into workflows, which then pass inputs to processes.
😐 1
So if you have enough params in your modules to be worrying about documentation for them, it might be worth considering whether any can be inputs instead..
a
I guess that would also create documentation for them, as they would be listed as inputs in the meta.yml. However, wouldn't this be more channels to handle when creating pipelines?
a
Yes, you get more channels but this is only really a 1-time “issue” when you first write the pipeline/module and the increased explicitness makes changes much easier to catch. Sarek is an example pipeline that has already done a lot of work extracting the use of global params from its modules/subworkflows. You can see it means passing in a lot of parameters explicitly to the top-level workflow but this makes it easier to follow later on imo
👍🏻 1
👍 1
s
I think that having 40+ individual parameters that must be passed in order just causes a different issue.
this is fine fire 3
🫠 1
p
You should be able to pass a map as an input..?
Having logic to handle 40+ options within a module is also its own issue 😅
💯 2
😬 2
a
Thanks guys! I’ll look into the sarek pipeline. I guess passing them to the top-level workflow is just different work then assigning them individually to each modules. And, just to clarify, we don’t have modules with 40+ parameters 😅 although some may have ~10 😬
a
@Alex Valcourt Caron here is our conversation about params and input 🙂 Thank you
a
I have been following the discussion on params in nf-core for a while now. I'm kind of disappointed no real solution to this has been proposed since. Also kind of confused all propositions here seem to have a big downside, some with high development burdens that make them unusable long term. Params in Input will be a pain to manage, more complex as module get added to more and more workflows and pipelines. Hiding them in a map makes linting and validation difficult and there is no guideline for it as of now. We can't use the param scope. Is there any thoughts about this right now ? Is using Input the final decision or will there be something better provided to do this job in the near future ?
p
What would your suggestion be for something better?
a
Hi @phil ! I'd love to have a straightforward answer for you (I think you would too), but I don't. There is no one-size-fits-all solution provided to us in the Nextflow DSL. I don't like Inputs as a replacement for params because I can see the management hurdles that we avoid using simple parameters, I'm sure I'm not the only one. We all don't want to use the
params
scope, because of the mess and potential collisions it'll create.
nf-core
and
nextflow
don't want us to use
task.ext
, because it is not really meant for it.
If I had to choose the best option for my team right now, it would be
task.ext
. It was about 2 years ago and still is today. It does all heavy lifting maintenance wise. Our users get used to them quite easily. The only problem we have had with it is documentation, but we found a way. We render it on a website and uses a new field in
meta.yml
. This could easily be added to
nf-core
commands and help users to have a full understanding of each module.
a
@Phil Ewels ⬆️ 😉
b
IMO the move from params to process/workflow inputs will be painful for large pipelines until we make it easier to pass many params without using a map or the
params
global I think the answer is record types. You bundle a bunch of related params into a record, pass the record around, and pass whatever record members you need into the process For this reason I haven't pushed the params issue very much (keeping it as a "paranoid" warning for now) until the language supports record types
a
In the meantime, I think it would be very easy to address the lack of documentation regarding the params in modules. • Add keyword
params
in meta.yml • Make sure
nf-core module info
can list these params • Make sure it can be linted
s
Something like this could be done, even if we aren't planning on using it within nf-core
a
Exactly this is just for "decoration" but it will make the life of many users easier 🙂
a
I'm happy to see there is some advancement to a workable solution on Nextflow's side. I don't have enough groovy knowledge to fully comprehend what record types will do, but I'm hopeful it could get us there. In any case, knowing the great @Ben Sherman is on the case is reassuring 😉 Are there any roadmaps we can follow to get a gist of how this issue gets worked ?
@Simon Pearce we can probably help each other and coordinate work to get this implemented quickly !
s
Just to be clear, I don't have any use for this myself, just giving my opinion 😉
But more people willing to contribute to the nf-core tooling are welcomed
a
@Ben Sherman do you have any prototype or example of what a record type looks like? To me I really don't see how it's much different to a map/simple object? FWIW my two cents are that the internal pipeline part of this issue is not really a problem - it's set it and forget it. (And I'd just use a map passed in as eg
val(tool_params)
and if more
val(tool2_params)
. To me the bigger issue is how you capture generic user-supplied params.
ext.args
is horrible because it can't easily be additive. At one stage I wanted to use nested params namespaced for the tool e.g.
--minimap2.kmer_size 3
but I believe that there are reasons to avoid nested params. One possibly over-engineered option is to define tool params in your top-level params as a json/yml input file which you parse into a validated map of allowed options using nf-schema.
b
This PR sketches out many of the language features I am thinking about, including record types:
Copy code
record SraParams {
    ena_metadata_fields         : String
    download_method             : DownloadMethod
    skip_fastq_download         : boolean
    dbgap_key                   : Path?
}
There is also some discussion here about how to improve on the ext args. Just keep in mind that there are a lot of other ideas in that PR that aren't related here. But I do welcome all feedback in the PR comments 😄
a
Hi ! @Ben Sherman @Phil Ewels While you guys are figuring out the future of the ext/params 😉, I would like to know if you will be interested in implementing the couple of suggestions I made earlier in the discussion and if you have someone who can do it ? • Allow keyword
params
in meta.yml • Make sure
nf-core module info
can list these params • Make sure it can be linted Thank you in advance 🙂
p
These are nf-core specific and not to do with Nextflow, so they should be discussed in #tools and probably by the maintainers team.
The thread got kind of massive, so maybe we can reboot into a new one
👍 1