The latest nextflow edge version shows the followi...
# modules
n
The latest nextflow edge version shows the following warning because of the
when
section in the nf-core modules:
Copy code
WARN: The `when` process section is deprecated -- use conditional logic in the calling workflow instead
I think it might be a good time to start removing these from the modules and thus pipelines should follow suit real soon this way
m
People are using it in fairly clever ways and I think would lament it's loss if it was taken out before it's not supported anymore.
💯 1
n
In what ways are people using it that won't be easy to migrate?
It's been known that it will be deprecated soon so I don't think we need to wait until full deprecation for this
m
party this 1
n
Hmm okay that's a good one indeed!
r
@Ben Sherman
I agree with Mahesh. It's nice last resort
n
I agree, but we would probably need some way to ignore the warning then, people will start complaining about it otherwise?
m
But we should get pipeline maintainers to implement solutions so
ext.when
is not needed. There's already an example for the one above
n
This could cause some issues with
.join
operators later in the pipeline though 🤔
m
Can you elaborate?
n
A lot of pipeline use the
failOnMismatch
option for the
join
operator to catch mishaps in the pipeline logic. By basically removing some samples in the middle of the pipeline execution, you will almost definitely cause mismatches because of the missing outputs
m
As long as both channels use the restricted set, it should be fine.
n
Yes indeed, but you can't guarantee that for every process. The user doing this would need to understand the pipeline logic for this.
But yeah I'm nitpicking 😅
I like the idea of skipping samples that way 😛
m
Well that's what we're here for. To help the developer write better patterns.
🙌 2
party this 2
u
The when block is a life saver in difficult situations where the batch size is very large, the pipeline is complex such as nf-core/mag, the compute has burnt a couple of thousand dollars, and the final report can be reached by excluding a couple of QC processes here and there. This level of granular control is not possible with sample exclusion at the beginning of the pipeline. Perhaps, we can advocate for a warning enable/disable flag for “when” similar to “configProcessNamesValidation”?
💯 1
cc. @Sateesh_Peri
m
This level of granular control is not possible with sample exclusion at the beginning of the pipeline.
I think it's possible for any pipeline, as long the relevant information can be in samplesheet form.
u
Do you mean by including an optional skip/when flag for each process against each sample in the sample sheet? I would say that’s a pretty expensive trade compared to the “when” block which is already there and working. It’s not even broken. Is there a reason for depreciation?
s
after a recent run where I found myself in a situation wanting to skip a process....i now vote to keep this feature which is quite handy
m
Presence/absence is enough. It really depends on what needs to be excluded though. I would claim the example is not complicated either. One reason for deprecation is that it splits the workflow logic from the script. In the case of process skipping, it's easier to see and know about if it's included as a parameter in the workflow.
u
Within nf-core pipelines we are not using this block to implement workflow logic. It is being used as a sort of a last resort in situations where a pipeline user can configure a process or a specific task to skip without changing any pipeline code. This level of flexibility will have to be built into the pipeline if we remove the when block. When a tool will fail for which sample is not always possible to test during pipeline development.
this 2
💯 1
b
I'm not in a hurry to remove the when block. I think people reach for it as a shortcut because operators are complicated to use and there aren't any good built-in mechanisms for error handling at the workflow level. If we can improve those aspects of workflow logic then I think it will be safe to remove
when
. A similar argument can be made for
each
. But the overall goal is to de-couple the process definition from the workflow logic, an ongoing effort since the introduction of DSL2
👍 2
m
I still think of it as workflow logic, and error handling is really something that does need to be improved. nf-core modules at this point don't really handle sample failure well which is why we end up using
errorStrategy = ignore
to work around it. A script error translates to process instance failure, but it would be nice to have a better mechanism to handle it without filling the process code with traps and exit code tests.
this 4
b
I agree. But I don't know how you would separate "domain errors" (e.g. failed sample) from "execution errors" (e.g. running out of memory/wall-time) without inspecting exit codes? Or do you think tasks should always return exit code 0 in the event of a domain error, and communicate the failure through the presence/absence of specific files?
s
I used
fastquorum
the other day to generate ubam files from fastqs with the UMIs included, turning off all the downstream processes. Saved me from writing a tiny pipeline to do the job, or from running the entire pipeline, or trying to kill it after it had done that process but not the rest. I've done that kind of hack several times in the past, to use pipelines not for their intended use, e.g. to check the logic on a csv input is working without it complaining that I don't actually have the data ready yet. Or to ignore a QC process that I'm not interested in (or isn't relevant), to save compute time or resources. If a pipeline has 10 QC steps, should we make 10 separate parameters to enable them to be skipped? Or the sarek way of a
skip_tools
parameter that then has to be parsed with lines like
params.skip_tools && params.skip_tools.split(',').contains('baserecalibrator')
What do I do about that one crappy tool that takes 12 hours when I give it a sample with 5x more reads than normal? Do I just have to wait for it? I guess I could write a custom process definition to give it no resources and
errorStrategy = 'ignore'
. It would be nice to just be able to directly say
Copy code
withName: 'FOO' { 
  enabled = false
}
Some workflows do indeed use
when
for workflow logic currently, including sarek (which also has the ability to opt in or out of many processes via parameters, with lots of fiddly code that is not as clean as just using when 😉 ). I don't know how you get rid of that without retaining some flexibility.
💥 1
party this 2
this 1
b
I think pipelines should just be more modular so that it's easier to run only certain parts of a workflow, or even a single process Write an entry workflow for every process and subworkflow in your pipeline (or ask Seqera AI to do it for you). It will serve both as documentation to others who use your module, and as an alternate entrypoint for you. Then maybe you don't need a hundred control-flow parameters in the top-level workflow Honestly I'd love to find a way to generate entry workflows with the language server, but in the meantime maybe Seqera AI can do a decent job
m
Hi. I just want to +1 what @Usman @Sateesh_Peri and @Simon Pearce said.
when
is very practical to quickly disable a pipeline step that we don’t need or would fail and terminate the whole workflow without having to change the workflow code (especially when using a 3rd-party pipeline)
👍 1
b
quick update • the deprecation warning for
when
was removed in Nextflow 25.04. it's still deprecated and the language server will flag it, but it won't bother pipeline users. we won't actually remove it until we have a good solution for the issues described in this thread • follow this issue for more discussion on custom error handling for processes. I just posted a proposed solution so feel free to share feedback
party this 4