Phil Ewels
07/05/2024, 8:59 PMworkflow.failOnIgnore
WhenI've seen this requested a few times in this Slack, so may be of interest to folks. Available from, the pipeline will exit with a non-zero exit code if any failed tasks are ignored using thetrueerror strategy.ignore
24.05.0-edge onwards. See https://www.nextflow.io/docs/latest/config.html#scope-workflowmoritz
07/05/2024, 9:01 PMmoritz
07/05/2024, 9:02 PMPhil Ewels
07/05/2024, 9:08 PMPhil Ewels
07/05/2024, 9:08 PMPhil Ewels
07/05/2024, 9:09 PMignore its relatively difficult to detect if there any problems or notJames Fellows Yates
07/06/2024, 4:32 AMPhil Ewels
07/06/2024, 5:03 AMJames Fellows Yates
07/06/2024, 7:07 AMSo you want to let your pipeline get as far as it possibly can (eg. finish) if a certain process goes wrong, but you also want to know about it.
James Fellows Yates
07/06/2024, 7:07 AMJames Fellows Yates
07/06/2024, 7:08 AMfail error strategy meansJames Fellows Yates
07/06/2024, 7:12 AMignore to allow the pipeline to continue
3. If you additionally set failOnIgnore this means that even if it's set to ignore, the pipeline to fail -> isn't this nr 1 again?James Fellows Yates
07/06/2024, 7:13 AMPhil Ewels
07/06/2024, 7:17 AMPhil Ewels
07/06/2024, 7:17 AMJames Fellows Yates
07/06/2024, 7:18 AMPhil Ewels
07/06/2024, 7:29 AMJames Fellows Yates
07/06/2024, 7:31 AMPhil Ewels
07/06/2024, 7:33 AMPhil Ewels
07/06/2024, 7:33 AMPhil Ewels
07/06/2024, 7:34 AMfinish error strategy π
Phil Ewels
07/06/2024, 7:34 AMPhil Ewels
07/06/2024, 7:34 AMJames Fellows Yates
07/06/2024, 7:35 AMJames Fellows Yates
07/06/2024, 7:35 AMJames Fellows Yates
07/06/2024, 7:35 AMAdam Talbot
07/06/2024, 1:33 PMmoritz
07/06/2024, 2:02 PMJames Fellows Yates
07/06/2024, 2:03 PMSimon Pearce
07/07/2024, 6:58 PMSimon Pearce
07/07/2024, 6:59 PMJames Fellows Yates
07/07/2024, 7:58 PMJonathan Manning
07/08/2024, 8:26 AMignoreThenFail, but I'm glad this is out now :-)Simon Pearce
07/08/2024, 8:41 AMSimon Pearce
07/08/2024, 8:42 AMignoreStrategy for everything?James Fellows Yates
07/08/2024, 8:42 AMJames Fellows Yates
07/08/2024, 8:42 AMSimon Pearce
07/08/2024, 8:43 AMSimon Pearce
07/08/2024, 8:43 AMSimon Pearce
07/08/2024, 8:43 AMSimon Pearce
07/08/2024, 8:44 AMJames Fellows Yates
07/08/2024, 8:44 AMignore errorStrategy then?James Fellows Yates
07/08/2024, 8:44 AMignore error stragegy`...Simon Pearce
07/08/2024, 8:45 AMJames Fellows Yates
07/08/2024, 8:47 AMignore on those processesJonathan Manning
07/08/2024, 9:10 AMignore means you pretend everything is OK, even if it isn't. With the new mechanism you can get as much done as possible using ignore (maybe after retries), but still register an overall failure at the end.Adam Talbot
07/08/2024, 9:20 AMparams.errorStrategy = "terminate"
workflow {
exit_code_channel = nextflow.Channel.of(0, 0, 0, 1, 0)
PROCESS_1(exit_code_channel)
PROCESS_2(PROCESS_1.out)
}
process PROCESS_1 {
errorStrategy "${params.errorStrategy}"
input:
val exitCode
output:
val exitCode
script:
"exit ${exitCode}"
stub:
"echo ${exitCode}"
}
process PROCESS_2 {
input:
val exitCode
output:
tuple val(exitCode), path('*.txt')
script:
"echo ${exitCode} > output.txt"
stub:
'touch output.txt'
}James Fellows Yates
07/08/2024, 9:24 AMAdam Talbot
07/08/2024, 9:24 AMworkflow scopeJames Fellows Yates
07/08/2024, 9:25 AMignore strategy does exactlyJames Fellows Yates
07/08/2024, 9:25 AMJames Fellows Yates
07/08/2024, 9:25 AMJames Fellows Yates
07/08/2024, 9:26 AMAdam Talbot
07/08/2024, 9:27 AMworkflow scope, but I didn't truly understand themJonathan Manning
07/08/2024, 9:27 AMJames Fellows Yates
07/08/2024, 9:29 AMJames Fellows Yates
07/08/2024, 9:29 AMJames Fellows Yates
07/08/2024, 9:29 AMJonathan Manning
07/08/2024, 9:30 AMAdam Talbot
07/08/2024, 9:30 AMβ’ terminate: the entire pipeline will fail and will not analyse any samples
β’ finish: Nextflow will finish any existing jobs but not submit any more
β’ ignore: 95 samples will be successfully analysed, 1 will fail and Nextflow will not raise the error
β’ ignore + workflow.failOnIgnore: Nextflow will successfully analyse the 95 samples, then exit with an error status and report the error.
James Fellows Yates
07/08/2024, 9:32 AMJames Fellows Yates
07/08/2024, 9:32 AMAdam Talbot
07/08/2024, 9:35 AMJames Fellows Yates
07/08/2024, 9:37 AMJames Fellows Yates
07/08/2024, 9:38 AMJonathan Manning
07/08/2024, 9:39 AMJames Fellows Yates
07/08/2024, 9:40 AMJames Fellows Yates
07/08/2024, 9:41 AMAdam Talbot
07/08/2024, 10:44 AMJonathan Manning
07/08/2024, 11:15 AMAdam Talbot
07/08/2024, 11:25 AMJonathan Manning
07/08/2024, 11:27 AMPhil Ewels
07/08/2024, 1:47 PMPhil Ewels
07/08/2024, 1:47 PMJames Fellows Yates
07/08/2024, 3:29 PMAdam Talbot
07/08/2024, 3:50 PMJames Fellows Yates
07/08/2024, 3:50 PMJames Fellows Yates
07/08/2024, 3:51 PMAdam Talbot
07/08/2024, 3:54 PMJames Fellows Yates
07/08/2024, 3:55 PMJames Fellows Yates
07/08/2024, 3:55 PMAdam Talbot
07/08/2024, 3:55 PMJames Fellows Yates
07/08/2024, 3:55 PMAdam Talbot
07/08/2024, 3:55 PMAdam Talbot
07/08/2024, 3:59 PMJames Fellows Yates
07/08/2024, 4:00 PMJonathan Manning
07/08/2024, 4:01 PMJames Fellows Yates
07/08/2024, 4:02 PMJames Fellows Yates
07/08/2024, 4:03 PMJonathan Manning
07/08/2024, 4:10 PMJames Fellows Yates
07/08/2024, 4:11 PMPhil Ewels
07/09/2024, 5:42 AMPhil Ewels
07/09/2024, 5:42 AMJonathan Manning
07/09/2024, 8:38 AMAdam Talbot
07/09/2024, 1:02 PMJames Fellows Yates
07/10/2024, 7:59 AM