Nicolas Vannieuwkerke
04/23/2025, 12:34 PMwhen section in the nf-core modules:
WARN: The `when` process section is deprecated -- use conditional logic in the calling workflow instead
I think it might be a good time to start removing these from the modules and thus pipelines should follow suit real soon this wayMahesh Binzer-Panchal
04/23/2025, 12:35 PMNicolas Vannieuwkerke
04/23/2025, 12:36 PMNicolas Vannieuwkerke
04/23/2025, 12:37 PMMahesh Binzer-Panchal
04/23/2025, 12:37 PMNicolas Vannieuwkerke
04/23/2025, 12:39 PMRike Hanssen
04/23/2025, 12:41 PMRike Hanssen
04/23/2025, 12:42 PMNicolas Vannieuwkerke
04/23/2025, 12:43 PMMahesh Binzer-Panchal
04/23/2025, 12:43 PMext.when is not needed. There's already an example for the one aboveNicolas Vannieuwkerke
04/23/2025, 12:55 PM.join operators later in the pipeline though 🤔Mahesh Binzer-Panchal
04/23/2025, 12:55 PMNicolas Vannieuwkerke
04/23/2025, 12:57 PMfailOnMismatch option for the join operator to catch mishaps in the pipeline logic. By basically removing some samples in the middle of the pipeline execution, you will almost definitely cause mismatches because of the missing outputsMahesh Binzer-Panchal
04/23/2025, 12:59 PMNicolas Vannieuwkerke
04/23/2025, 1:00 PMNicolas Vannieuwkerke
04/23/2025, 1:00 PMNicolas Vannieuwkerke
04/23/2025, 1:00 PMMahesh Binzer-Panchal
04/23/2025, 1:01 PMUsman
04/23/2025, 3:33 PMUsman
04/23/2025, 3:35 PMMahesh Binzer-Panchal
04/23/2025, 6:16 PMThis level of granular control is not possible with sample exclusion at the beginning of the pipeline.I think it's possible for any pipeline, as long the relevant information can be in samplesheet form.
Usman
04/23/2025, 6:46 PMSateesh_Peri
04/23/2025, 7:02 PMMahesh Binzer-Panchal
04/23/2025, 7:14 PMUsman
04/23/2025, 7:24 PMBen Sherman
04/23/2025, 7:33 PMwhen. A similar argument can be made for each.
But the overall goal is to de-couple the process definition from the workflow logic, an ongoing effort since the introduction of DSL2Mahesh Binzer-Panchal
04/23/2025, 8:12 PMerrorStrategy = ignore to work around it. A script error translates to process instance failure, but it would be nice to have a better mechanism to handle it without filling the process code with traps and exit code tests.Ben Sherman
04/23/2025, 9:18 PMSimon Pearce
04/24/2025, 5:48 AMfastquorum the other day to generate ubam files from fastqs with the UMIs included, turning off all the downstream processes. Saved me from writing a tiny pipeline to do the job, or from running the entire pipeline, or trying to kill it after it had done that process but not the rest.
I've done that kind of hack several times in the past, to use pipelines not for their intended use, e.g. to check the logic on a csv input is working without it complaining that I don't actually have the data ready yet.
Or to ignore a QC process that I'm not interested in (or isn't relevant), to save compute time or resources.
If a pipeline has 10 QC steps, should we make 10 separate parameters to enable them to be skipped? Or the sarek way of a skip_tools parameter that then has to be parsed with lines like params.skip_tools && params.skip_tools.split(',').contains('baserecalibrator')
What do I do about that one crappy tool that takes 12 hours when I give it a sample with 5x more reads than normal? Do I just have to wait for it? I guess I could write a custom process definition to give it no resources and errorStrategy = 'ignore' . It would be nice to just be able to directly say
withName: 'FOO' {
enabled = false
}
Some workflows do indeed use when for workflow logic currently, including sarek (which also has the ability to opt in or out of many processes via parameters, with lots of fiddly code that is not as clean as just using when 😉 ). I don't know how you get rid of that without retaining some flexibility.Ben Sherman
04/24/2025, 1:25 PMMatthieu Muffato
04/25/2025, 10:30 AMwhen is very practical to quickly disable a pipeline step that we don’t need or would fail and terminate the whole workflow without having to change the workflow code (especially when using a 3rd-party pipeline)Ben Sherman
06/02/2025, 10:56 PMwhen was removed in Nextflow 25.04. it's still deprecated and the language server will flag it, but it won't bother pipeline users. we won't actually remove it until we have a good solution for the issues described in this thread
• follow this issue for more discussion on custom error handling for processes. I just posted a proposed solution so feel free to share feedback