Arnaud Boré
02/04/2025, 4:51 PMparams for our users. We've been developing a lot of modules in our library (nf-neuro) for our users to help them implement their own neuroimaging pipeline.
It is becoming obvious that without any documentation about the params used in the module, our users are lost and can't easily use our modules.
It also became obvious that the topic params is far from being resolved.
My suggestion was to, in the meantime, implement these useful features...
• Allow keyword params in meta.yml
• Make sure nf-core module info can list these params
• Make sure it can be linted
Is there something that some dev can push forward ?
Thank you in advance 🙂James Fellows Yates
02/04/2025, 7:29 PMJames Fellows Yates
02/04/2025, 7:30 PMJames Fellows Yates
02/04/2025, 7:30 PMJames Fellows Yates
02/04/2025, 7:32 PMArnaud Boré
02/04/2025, 7:35 PMArnaud Boré
02/04/2025, 7:35 PMJames Fellows Yates
02/04/2025, 7:36 PMJames Fellows Yates
02/04/2025, 7:36 PMJames Fellows Yates
02/04/2025, 7:37 PMArnaud Boré
02/04/2025, 7:37 PMJames Fellows Yates
02/04/2025, 7:38 PMJames Fellows Yates
02/04/2025, 7:38 PMArnaud Boré
02/04/2025, 7:39 PMJames Fellows Yates
02/04/2025, 7:39 PMArnaud Boré
02/04/2025, 7:39 PMJames Fellows Yates
02/04/2025, 7:40 PMJames Fellows Yates
02/04/2025, 7:40 PMJames Fellows Yates
02/04/2025, 7:40 PMArnaud Boré
02/04/2025, 7:40 PMJames Fellows Yates
02/04/2025, 7:41 PMJames Fellows Yates
02/04/2025, 7:41 PMJames Fellows Yates
02/04/2025, 7:41 PMArnaud Boré
02/04/2025, 7:45 PMArthur
02/04/2025, 9:00 PMext.arg also seems a bit against the nf-core standard where the number of tools should really only be 1/2.
I would say that it's better to stop using the term params for anything other than the nextflow params object/scope because as James says it causes confusion. I don't like args either really, would prefer tool_args or tool_params.
When I look at one of your modules I see you are using specific terms in ext. which nf-core would typically use input values for insteadArnaud Boré
02/04/2025, 9:07 PMArthur
02/04/2025, 9:19 PMext.<> is not how nf-core does this. In nf-core those specific values would be expected to be declared as task inputs (which are documented in the meta yml). ext.args is then just a catch-all for any non-required tuning parameters people may want to use.
I think nf-core starting to support a second method of doing the same job would be confusing and create inconsistency across modules.Arnaud Boré
02/05/2025, 12:01 AMArthur
02/05/2025, 12:12 AMext.args. There is no need to designate each one separately?
This to me just seems to add unnecessary complexity and creates a difference between the way users of nf-core modules would expect the code to behave.
In your module I linked above for example, you have mirrored each param for the tool explicitly except for --mask (obvious reason as it is a path) and -v for the verbose logging.
I am unclear why:
process {
withName: IMAGE_POWDERAVERAGE {
ext.args = "-v --shell_thr 10"
}
is considered worse than:
process {
withName: IMAGE_POWDERAVERAGE {
ext.args = "-v"
ext.shell_thr = 10
}Arnaud Boré
02/05/2025, 12:17 AMArnaud Boré
02/05/2025, 12:18 AMArthur
02/05/2025, 12:24 AMAlso the module you choose has only one main command but a lot of module tool_args are used for different set of commandsI don’t quite follow this, perhaps you can show an example
Arnaud Boré
02/05/2025, 1:18 AMArthur
02/05/2025, 1:25 AMdef args = task.ext.args ?: ""
def args2 = task.ext.args2 ?: ""
def args3 = task.ext.args3 ?: ""
Followed by:
then
scil_dwi_extract_b0.py $image $bval $bvec ${prefix}__b0.nii.gz --mean \
$args --skip_b0_check
bet ${prefix}__b0.nii.gz ${prefix}__image_bet.nii.gz -m -R $args2
scil_volume_math.py convert ${prefix}__image_bet_mask.nii.gz ${prefix}__image_bet_mask.nii.gz --data_type uint8 -f
mrcalc $image ${prefix}__image_bet_mask.nii.gz -mult ${prefix}__image_bet.nii.gz -quiet -nthreads 1 -force
else
bet $image ${prefix}__image_bet.nii.gz -m -R $args2
scil_volume_math.py convert ${prefix}__image_bet_mask.nii.gz ${prefix}__image_bet_mask.nii.gz --data_type uint8 -f
fi
if [ "$task.ext.crop" = "true" ];
then
scil_volume_crop.py ${prefix}__image_bet.nii.gz ${prefix}__image_bet.nii.gz -f \
--output_bbox ${prefix}__image_boundingBox.pkl
scil_volume_crop.py ${prefix}__image_bet_mask.nii.gz ${prefix}__image_bet_mask.nii.gz -f\
--input_bbox ${prefix}__image_boundingBox.pkl
scil_volume_math.py convert ${prefix}__image_bet_mask.nii.gz ${prefix}__image_bet_mask.nii.gz \
--data_type uint8 -f
fi
if [ "$task.ext.dilate" = "true" ];
then
scil_volume_math.py dilation ${prefix}__image_bet_mask.nii.gz $args3 ${prefix}__image_bet_mask.nii.gz --data_type uint8 -f
fi
If they were shared across different tools then I would probably say that they should become real inputs.Arnaud Boré
02/05/2025, 1:33 AMbet_f and not if the same brain would have been acquired with a SIEMENS scanner. Those are realistic use cases in neuroimaging.Arthur
02/05/2025, 1:42 AMbet tool presumably has it’s own documentation somewhere and the users should be interrogating that.
The part here I would say (and it’s something I think myself) is that ext.args / ext.args2 is not intuitive or informative as to which tool the args will be piped to. However I also understand why it’s standardised across nf-core as it makes it easier to remember or predict syntax, particularly in the case of a single tool (i.e. I can always do ext.args = … and I don’t have to think if the module uses e.g. ext.picard or ext.Picard )
In that sense I would support some mapping of the generic args to a tool in the meta.yml
e.g.:
ext:
args: samtools merge
args2: samtools indexAnthony Gagnon
02/05/2025, 2:09 AMext.args{1,2,...} for modules containing a single command, as you indeed can refer to the tool's documentation. However, and I think this is where we mostly differ from nf-core philosophy, is that most of our modules are actually a combination of tools (in the example you cited, there are 3 tools: bet , mrcalc, and scil_volume_crop. This is a decision we made, knowing it was not nf-core philosophy, to reduce the number of files we generate (we often launch our pipelines on 1000+ subjects, which can easily and quickly exceed a cluster's quota for the number of files).
This led us to name the ext to facilitate the understanding of which ext was associated with which tool. If we take the example you linked, the three tools come from three neuroimaging libraries. Does this mean the users should refer to three documentation sites for a single module? To me, this doesn't really make sense and represents a burden for the user. Documentation for those types of `ext`` is a must-have in our case.
Thinking out loud here (if that's possible by writing...), I guess this stems from how we designed our modules. To us, they represent an "operation" applied to an image, which can require multiple tools, whereas, in the nf-core philosophy (AFAIK), the modules are tools. Does this makes sense?Arthur
02/05/2025, 2:17 AMext.args to ext.bet you are splitting what would go into ext.args into many different ext.<> for each possible CLI arg for each tool.
But as I have said I do think ext.args alone is quite opaque and in fact nf-core modules which use multiple argsN such as bcftools/mpileup are not always easy to guess what command any given argsN will be passed to, so I would support some way to document that.Anthony Gagnon
02/05/2025, 2:40 AMAnthony Gagnon
02/05/2025, 2:44 AMYou aren’t just renaming e.g.This made me realize the advantages of doing it the waytoext.argsyou are splitting what would go intoext.betinto many differentext.argsfor each possible CLI arg for each tool.ext.<>
nf-core is doing it now (I guess it's better late than never 😅). I would have to test it out in our modules, but it does relieve the burden of supporting every possible CLI args the tools have and makes updating versions way more manageable when there are breaking changes.James Fellows Yates
02/05/2025, 8:14 AMext.args2/3/4 into their own section with better metadata. But renaming them to non-standardised stuff is extremely unlikely to happen.
I think it would be best to make an issue on #tools if you've not already, and we can discuss with Júlia and Matthias the best way to do this.Arthur
02/05/2025, 8:22 AMargs into the meta.yml is that is can be made as a non-breaking change and/or is much, much easier to backwards update all the existing modules forJames Fellows Yates
02/05/2025, 8:25 AMJames Fellows Yates
02/05/2025, 8:26 AM- "bcftools plugin setGT":
description: "Bcftools plugins are tools that can be used with bcftools to manipulate
variant calls in Variant Call Format (VCF) and BCF. The setGT plugin sets genotypes according to the specified criteria and filtering expressions. For example, missing genotypes can be set to ref, but much more than that."
homepage: "https://samtools.github.io/bcftools/howtos/plugins.html"
documentation: "https://samtools.github.io/bcftools/howtos/plugin.setGT.html"
tool_dev_url: "https://github.com/samtools/bcftools"
doi: "10.1093/bioinformatics/btp352"
licence: ["MIT"]
args_id: "$args2"
identifier: biotools:bcftools
Because IMO this is actually sufficientJames Fellows Yates
02/05/2025, 8:26 AMargs2 is compeltely arbitraryJames Fellows Yates
02/05/2025, 8:26 AMJames Fellows Yates
02/05/2025, 8:27 AMbcftools/pluginsetgt btwArthur
02/05/2025, 8:37 AMargs_id already exists lol. That’s v similar to what I was proposingArthur
02/05/2025, 8:40 AMmpileup is listed (and doesn’t have an args_id), but yet:
bcftools \\
mpileup \\
--fasta-ref $fasta \\
$args \\
$bam \\
$intervals \\
$mpileup \\
| bcftools call --output-type v $args2 \\
| bcftools reheader --samples sample_name.list \\
| bcftools view --output-file ${prefix}.vcf.gz --output-type z $args3Arthur
02/05/2025, 8:42 AMargs_id from the tools section is useful because it allows for cases where also there is second invocation of the same tool (not sure how many cases this is but would not be shocked if it’s non-zero).
Not real example which hopefully demonstrates how this is different:
args_ids:
args: samtools index
args2: samtools merge
args3: samtools index second invocationJames Fellows Yates
02/05/2025, 8:43 AMJames Fellows Yates
02/05/2025, 8:43 AMJames Fellows Yates
02/05/2025, 8:43 AMOh. How did I not notice args_id already exists lol. That’s v similar to what I was proposingI swear it was added as a requirement recently and then now is not linted for...
James Fellows Yates
02/05/2025, 8:44 AMJames Fellows Yates
02/05/2025, 8:45 AMinput sectionsArthur
02/05/2025, 8:47 AMJames Fellows Yates
02/05/2025, 8:51 AMJames Fellows Yates
02/05/2025, 8:51 AMArthur
02/05/2025, 8:52 AMJames Fellows Yates
02/05/2025, 8:52 AMJames Fellows Yates
02/05/2025, 8:53 AMArthur
02/05/2025, 8:57 AMext.args. So would be really cool to be able to display a list of qualified module names and what their ext.args control, but I think that could be quite convolutedJúlia Mir
02/05/2025, 9:00 AMmeta.yml to describe the ext.args values. At the level of tools , inputs and outputs .
Sounds good! I think we only need a description for those. So it would be something like
tools:
- "bcftools":
description: "..."
homepage: ...
args_id: "$args"
identifier: biotools:bcftools
- "bcftools plugin setGT":
description: "..."
homepage: ...
args_id: "$args2"
identifier: biotools:bcftools
args:
- args:
description: ""
- args2:
description: ""
input:
- - meta:
...
- vcf:
...James Fellows Yates
02/05/2025, 9:01 AMdescription and 'subcommand' attribute would be perfectArthur
02/05/2025, 9:03 AMJames Fellows Yates
02/05/2025, 9:03 AMJames Fellows Yates
02/05/2025, 9:03 AMJames Fellows Yates
02/05/2025, 9:03 AMsamtools sortJames Fellows Yates
02/05/2025, 9:03 AMsamtools viewJames Fellows Yates
02/05/2025, 9:04 AMJúlia Mir
02/05/2025, 9:05 AMargs_id then?James Fellows Yates
02/05/2025, 9:05 AMargs_id doesn't sa the subcommandJames Fellows Yates
02/05/2025, 9:05 AMbcftoolsJames Fellows Yates
02/05/2025, 9:05 AMsamtoolsJames Fellows Yates
02/05/2025, 9:05 AMsamtools view then samtools sort etc.Júlia Mir
02/05/2025, 9:06 AMargs_id is only useful if there are more than one tools, got itArthur
02/05/2025, 9:07 AMtools will list or link to a sub command. It might just be the top-level tool and then it's less informativeSimon Pearce
02/05/2025, 9:53 AMargs, args2 policy, and do think it is much more helpful to name them with respect to what they actually do.
I like Júlia's suggestion of a separate entry in the yml, and think that description would be sufficient.Arnaud Boré
02/05/2025, 2:10 PMJames Fellows Yates
02/05/2025, 2:54 PMJames Fellows Yates
02/05/2025, 2:55 PMArnaud Boré
02/05/2025, 3:07 PM