https://nf-co.re logo
Join Slack
Powered by
Nextflow v26.04.0 uses the strict syntax parser by...
# help
n
Nextflow v26.04.0 uses the strict syntax parser by default and this does break a lot. But I think one of the lesser known changes is that the automatic type conversion of parameters passed via the command line has been removed. All parameters passed via the CLI will be of the 
String
type by default. This will be solved by static typing introduced in Nextflow 26.10 but there is currently no built-in Nextflow way to handle this properly. You can manually update the types of the parameters by doing something like this: 
params.test_parameter = params.test_parameter as Integer
to turn a parameter to a specific type (Integer in this case). Before releasing your pipeline as strict syntax compatible, please double check that it also works when changing parameters via the CLI (as types are preserved when specified in the config or parameters file)
😬 2
this 2
m
ufffff. should we add a linting test for that in nf-core tools? Or a github action in the template?
n
I'm not sure how to best approach this as it is a temporary issue 😕
I'm running some tests on my pipeline now to see if there is an easy fix
m
temporary as in "solved in half a year"?
100 rainbow 1
n
fair
How would we lint for this?
m
we can check for non string types in the 
nextflow_schema.json
and do a quick parsing for a type casting to the right type in any of the code
👍🏻 1
n
Sounds good, ideally these should be done above the entry workflow too so that should make linting for it a bit easier
m
what do you mean exactly with "above the entry workflow"? Do we only need to check 
./main.nf
?
n
Copy code
params.igenomes_ignore = params.igenomes_ignore != null ? params.igenomes_ignore as Boolean : null

workflow {
...
}
yes!
when static typing will be introduced all parameter type casting will also be in that location
👍🏻 1
this 1
An example of my implementation: https://github.com/nf-cmgg/smallvariants/pull/328
Scratch that, it doesn't work. I'll need to rethink this
m
cc @Phil Ewels if you have ideas on how to handle this.
n
I'll try writing a plugin which we can just use as an opt-in feature 🙂
👍🏻 1
p
> until Nextflow supports static typing (26.10+) But it supports it already, no?
In fact I thought that we had typed params since 25.10 already
n
Yes but it's not a stable feature yet
p
Yeah it was meant to be, some last minute cold-feet meant that the flag stayed in if that's what you mean
But wait, are you sure params aren't stable? 🤔 Getting mixed up with the different parts of the type system
@Ben Sherman @Chris Hakkaart @Jorge Ejarque can anyone help me out on this thread please? 😅
n
It still shows a warning about typing that it's a preview feature: https://docs.seqera.io/nextflow/process-typed
But wait, are you sure params aren't stable? 🤔 Getting mixed up with the different parts of the type system
The problem only occurs when using strict syntax and passing a parameter via the CLI, these will always be of the 
String
type now instead of having their type derived (as was the case in the old syntax)
I'm already seeing some people posting issues about this, I can share some links if you want to
p
That link is for processes, not params
Processes are still in preview, as are workflows 👍🏻
n
Well that solves the issue, but it should be checked nonetheless
@mashehu you can do linting for that immediately then 🙂
p
You found it! I was looking for that page everywhere lolsob
Forgot that params were on the workflows page
@Chris Hakkaart I find this docs layout super confusing.. What do you think about having Params as its own page under "Developing pipelines"? (can merge in some of the other pages we have about params)
party this 1
👀 1
n
I'll give this a go now in my pipeline 😉
For future reference, running 
Nextflow: Convert script to static types
on 
<http://main.nf|main.nf>
in VScode automatically puts the typed params block in that file
this 1
p
Yes I want to lean into this more in the coming weeks and see if we can make this conversion tooling more accessible
I was also wondering about doing a PR blitz across nf-core pipelines with these changes too 🤔 But haven't fully considered what that'd look like yet
m
i am assuming this is not part of 
nextflow lint
?
p
I don't think so.. I was discussing all this with @Ben Sherman a bit last week, not sure if he's awake yet and has thoughts on the topic..?
m
having to go through an editor is sub-optimal
p
For sure 💯
n
Yeah I agree! I'm also still having some issues getting the pipeline to actually run
s
Linking some relevant GH issues / prs from a month or so ago that I think might be useful for the discussion: • Boolean flag default to true set to false through CLI is interpreted as a string nextflow-io/nextflow#6924 • Using strict syntax causes template to raise an error when setting boolean options on CLI nf-core/tools#4100 • Cast strict syntax parameters to types nextflow-io/nf-schema#208
On the problem in the parent message, Ben recommended this approach from the first link out above (which aligns with consensus in the thread via typed parameters as far as I can tell): nextflow.config
Copy code
params {
  boolean_flag = false
}
main.nf
Copy code
params {
  boolean_flag: Boolean
}

workflow {
  println "boolean_flag: ${params.boolean_flag} (${params.boolean_flag.getClass()})"
}
👍🏻 1
n
Yes using nf-schema 2.7.2 will fix the issue you had 🙂. It's even better to use typed parameters as pointed out above in this thread
❤️ 1
Okay I got it fully working: https://github.com/nf-cmgg/smallvariants/pull/328 Still having some issues with nf-core lint though:
Copy code
│ nextflow_config: Config variable not found: params.input                                                                                                                                                                                             │
│ nextflow_config: Default value from the Nextflow schema params.genome = GRCh38`` not found in nextflow.config.                                                                                                                                       │
...
│ schema_params: Default value for param input invalid: Not in pipeline parameters. Check nextflow.config.                                                                                                                                             │
...
m
we don't support these kinds of parameter definitions yet. for now you need to use
Copy code
lint:
  nextflow_config:
    - params.input
i guess
n
I have about 100 lines like that in my lint output, I will probably just ignore those errors I think
m
how about
Copy code
lint:
  nextflow_config: false
n
That's also possible 😛
I might do that temporarily then
👍🏻 1
All dependencies (except for nf-core lint) have now been updated to make it possible to use parameter types 🙂
🙌🏻 2
p
Great work!
We should probably start to think about what we want for conventions, eg. 
<http://main.nf|main.nf>
- should we do inline comments? Currently 3 lines per param which is pretty long. But maybe the formatter deletes those comments?
but.. do we need them? As it should be in the schema right? And in VSCode hovering on the param name should show the param info from the schema
(could potentially even update the Nextflow VSCode plugin to show that info inline as a "fake comment" kind of thing)
n
the plugin added those comments for me and I didn't bother deleting them but I agree that we should only have those in one place (the schema) with hover hints
m
no hints there yet
😪 1
👍 1
p
I'm sure it used to.. I remember doing recordings / screenshots of it.
n
I'm seeing some issues with igenomes. Apparently the 
genomes
map isn't set in the workflow when using parameter types. Looking into possible fixes now but just so you know
Problem seems to be caused by the fact that we use nested parameters for that. When I converted the igenomes.config file to be like this, it worked:
Copy code
params.genomes = [
    GRCh38: [
        fasta: "path/to/fasta",
        fai: "path/to/fai",
    ]
]
p
Right.. facepalm
I'm surprised this stuff didn't come up sooner.. 👀 Haven't we been sort of testing strict syntax since the start of the year? 😅
Maybe not the CLI stuff so much I guess..
n
Yeah AFAIK no one has actually properly tested parameter types until now 😁
at least not in an nf-core setting
But that should be a pretty easy fix as all igenomes config files are part of the pipelines, a template update should be enough
p
True 👍🏻
@mashehu I think this entire thread is very relevant to our chat yesterday
💯 1
🎉 1
n
Also: seems like the 
process.errorStrategy
config options does not allow dynamic setting anymore: 
_java.lang.IllegalStateException_: Directive 'errorStrategy' doesn't support dynamic value (or context not yet initialized)
p
hmm, that's annoying - and worth a GitHub issue I think. Are you ok to spin one up please? 🙏🏻
n
I'm doing some checks first that it's not caused by 
or context not yet initialized
, if not I will be happy to create an issue
thankyou 1
the input tuple of the process was wrong 😁 thanks strict syntax
Another side effect of this (not sure if I would call it something bad) is that nf-test doesn't use the defaults of the parameters anymore, which forces people to not use any parameters in workflows 😛
Still some issues in nf-test with typed parameters, I'll try and fix this asap
NVM, false alarm
phew obama 1
b
Added some guidance to the strict syntax docs on legacy vs typed parameters: https://docs.seqera.io/nextflow/strict-syntax#using-legacy-parameter-declarations
❤️ 3
p
For what it's worth; I just started running into this issue (i.e. when passing true/false values via a params.yml, true is seen as a string and triggers validation error 
Value is [string] but should be [boolean] nf-core
) after updating my nf-core template pipeline from 3.x to 4.0.2. I had been using using NFX_VER=26.04.1 up until now without issue. Downgrading to 25.10.4 resolves the problem. I only do these updates once in a blue moon and always end up terribly confused on which part of the ecosystem is acting up; both nextflow and nf-core are moving at such a rapid pace and changing their internals or sunsetting previous behaviour; it's difficult to keep up.
b
Indeed. We decided to introduce things piecemeal this time to avoid the monolithic shift of DSL1 -> DSL2. But the downside here is that the constant drip of changes feels chaotic For what it's worth, I think 26.04 represents the end of major changes for a while. I'm sure we will continue to improve things on the margins, but we have effectively completed what would have been "DSL3"
👍 1
7 Views
Open in Slack
PreviousNext
Powered by