In case you missed it, Nextflow just got a new con...
# random
p
In case you missed it, Nextflow just got a new config option:
workflow.failOnIgnore
When
true
, the pipeline will exit with a non-zero exit code if any failed tasks are ignored using the
ignore
error strategy.
I've seen this requested a few times in this Slack, so may be of interest to folks. Available from
24.05.0-edge
onwards. See https://www.nextflow.io/docs/latest/config.html#scope-workflow
❀️ 2
m
I don't get it, isn't the ignore strategy there to do the opposite? So if you don't want that, you just, you know, don't ignore the error? πŸ˜…
Or does this mean it silently continues but exits non-zero when the pipeline completed?
p
Exactly, the second thing ☝🏻
So you want to let your pipeline get as far as it possibly can (eg. finish) if a certain process goes wrong, but you also want to know about it.
❀️ 2
With just
ignore
its relatively difficult to detect if there any problems or not
j
Yeah I would argue the current docs aren't really clear on that with the current phrasing
p
Which? Any suggestions for improvements?
j
So you want to let your pipeline get as far as it possibly can (eg. finish) if a certain process goes wrong, but you also want to know about it.
Literally what you say
Basically to distinguish with the current
fail
error strategy means
It seems sort of counter intuitive currently: 1. Default behaviour of a nextflow run is for the pipeline to fail (presumably with exit code 1) if a task fails 2. You can set an erorrStrategy
ignore
to allow the pipeline to continue 3. If you additionally set
failOnIgnore
this means that even if it's set to ignore, the pipeline to fail -> isn't this nr 1 again?
The main thing to distinguish, as you and Moritz have said is that in 1. The pipeline will not run to the end/as far as it can before failing. It will stop any new tasks being executed immediately and then fail In 3. It will allow the rest of the runs task to executed before failing
p
Ok I'm on holiday now but will try to find time to put in a PR at some point.
Feel free to put something in if anyone has time. It's just markdown in the Nextflow repo
j
Did I summarise the difference correctly?
p
Yup, additions: 1. (Immediately) 2. Yup 3. It'll exit code 0 at the end
j
I thought with 2. If you allow ignoring of errors it will exit 0? And then the difference with 1/3 is just when exit 1 is reported?
p
Correct
With 3 it'll finish as much of the pipeline as it can. With 1 it'll exit immediately.
There's also
finish
error strategy πŸ˜…
Maybe we need a table / list
this 2
I thought it was pretty clear already tbh, but there are quite a few variations
this 1
j
Yeah increasingly sort of overlapping but not exactly....
I might make an issue instead
I don't think I'm familiar enough with all the options
πŸ‘πŸ» 1
a
This is one of those things where if you’ve never encountered the situation it probably makes no sense but for a whole group of people it’s extremely important. You should see the absurdly convolute code I’ve had to write to implement this. Heres the original ticket https://github.com/nextflow-io/nextflow/issues/4446 Perhaps we could phrase it with an example in the docs? I am analysing 96 patient samples, however one is corrupted and will fail. Here is the behaviour with the following strategies: fail: the entire pipeline will fail and will not analyse any samples ignore: 95 samples will be successfully analysed, 1 will fail and Nextflow will not raise the error failOnIgnore: Nextflow will successfully analyse the 95 samples, then exit with an error status and report the error.
πŸ‘ 1
m
Yes, it makes a lot of sense to me now. Basically, I used to run nextflow one sample at a time for this reason and let SLURM handle the resource distribution.
j
Yes that also works πŸ‘πŸ‘
s
I'm currently using a cluster which features somewhat random failures, so this is a much better plan than my current one of set every job to retry no matter what the error code
(but also will fix the cases in production where one patient sample failing kills the whole batch; at least once I convince them to swap to Nextflow...)
πŸŽ‰ 2
j
I made a teeny-weeny PR here to start discussion: https://github.com/nextflow-io/nextflow/pull/5119
😡 1
j
Some context on the origins of this: https://github.com/nextflow-io/nextflow/issues/4446 https://github.com/nextflow-io/nextflow/pull/5071 https://github.com/nextflow-io/nextflow/pull/4686 I preferred the proposal to have it as an errorStrategy
ignoreThenFail
, but I'm glad this is out now :-)
πŸ‘ 1
s
Uhh, that doesn't do what I was thinking
So we have to set
ignoreStrategy
for everything?
j
If I understand it correctly, yes you would (or on a per-process basis)
But maybe I don't understand it 🀷
s
I had assumed it had been implemented as Jonathan said, temporarily ignore fails then fail at the end
I mean, it makes sense to have both I guess
to show that ignored tasks actually were failed
I have dealt with dodgy tools that intermittantly failed that you want to know that they did
j
Maybe it's not related to the
ignore
errorStrategy then?
But it expliticly states `using the
ignore
error stragegy`...
s
Yeah, I think if you set this without setting anything to ignore it'll have no effect at all
πŸ‘ 1
πŸ‘πŸ» 1
j
So I would use it to allow failing of likely 'expected' failures anyway, and in which case, I would only set
ignore
on those processes
j
The point is basically the workflow's exit code.
ignore
means you pretend everything is OK, even if it isn't. With the new mechanism you can get as much done as possible using
ignore
(maybe after retries), but still register an overall failure at the end.
this 1
πŸ‘ 1
a
Copy code
params.errorStrategy = "terminate"

workflow {
    exit_code_channel = nextflow.Channel.of(0, 0, 0, 1, 0)
    PROCESS_1(exit_code_channel)
    PROCESS_2(PROCESS_1.out)
}

process PROCESS_1 {

    errorStrategy "${params.errorStrategy}"

    input:
    val exitCode

    output:
    val exitCode

    script:
    "exit ${exitCode}"

    stub:
    "echo ${exitCode}"
}

process PROCESS_2 {
    input:
    val exitCode

    output:
    tuple val(exitCode), path('*.txt')

    script:
    "echo ${exitCode} > output.txt"

    stub:
    'touch output.txt'
}
j
Yeah I don't think this is clear from the docs
a
Hmm I don't like it being under the
workflow
scope
this 1
j
Unless you implicitly know what the
ignore
strategy does exactly
Feel free to leave comments on my PR
Paolo replied but not very usefully 🀣
(as his comment isn't solving the issue 😬), but I made a new suggestion now in a comment that may partly address your point too @Adam Talbot (in a workaround but still)
a
I think there were technical reasons for it going under the
workflow
scope, but I didn't truly understand them
this 1
j
It made much more sense to me when it was just a different errorStrategy, but we're just mortals
j
Yeah, but I guess the strategy is on a per process basis but in this case you need to sum across all errors
this 1
That's why my new suggestion is to link from the erorrStrategy docs to that new option doxs
But say why you would use that option next to the erorrStrategy
j
Sounds like a plan
a
Something about how Nextflow handles the error status under the hood. Still, I think an example in the docs will go some way to helping. From my previous example:
β€’ terminate: the entire pipeline will fail and will not analyse any samples
β€’ finish: Nextflow will finish any existing jobs but not submit any more
β€’ ignore: 95 samples will be successfully analysed, 1 will fail and Nextflow will not raise the error
β€’ ignore + workflow.failOnIgnore: Nextflow will successfully analyse the 95 samples, then exit with an error status and report the error.
πŸ‘ 1
j
noice
now do it
a
in a minute
j
faster
(just told Paolo you'll do it too ;) )
j
Wait, we can do that? Who's James' big boss and what can we make him do?
j
Shes on holiday so SYKE!
😁 1
sike
j
Would have commented, but Paolo insta-merged πŸ˜„
🀣 2
a
Just open a new PR, Nextflow docs are just md
j
Nah, no insights worthy of that
p
Need a new PR to link the two sections of the docs anyway
Can do additional tweaks in that PR πŸ˜‰
j
a
REJECT
j
Given the links are broken and I don't understand it works in that framework,...
no surprise
a
I made some comments
j
I thought it was a reject 😏
(fanks tho)
a
Verify it works 😬
j
salute
a
I'm helping
it's working
j
yaaay
j
I'm going to add some suggestions to Adam's bit- give me a sec before merge...
j
Go ahead!
Tell Paolo to not merge πŸ˜‰
j
Done! Just felt it was a bit wordy, feel free to disregard if you don't agree
j
That works for me, if @Adam Talbot is OK with it?
πŸ‘ 1
p
Nice, thanks all! Yeah it's rendered with MYST which has weird linking syntax shakefist
Will hopefully change this before very long
πŸ‘ 1
j
Thanks for raising https://github.com/nextflow-io/nextflow/pull/5123 @Adam Talbot, wasn't being lazy, just thought James could smuggle it in with what he was doing πŸ˜‰
πŸ₯· 1
a
Always good to keep PRs discrete!
j
Fixed the conflict then, If someone can merge?