Hi Experts, We are running nextflow pipeline on GC...
# random
l
Hi Experts, We are running nextflow pipeline on GCP and we tried to use spot instance and noticed although in the workflow we specified retry for a few exit code, but nothing got a retry and our workflow very often fail due to spot instance preemption. Here’s some of the information:
nextflow 24.10
release this is in the .nf file for each process:
errorStrategy { task.exitStatus == null || task.exitStatus in [30,137,247,50001] ? (task.attempt >= 3 ? process.errorStrategy : 'retry') : process.errorStrategy }
unfortunately, we don’t see any thing got a retry and a common failure look like this:
Caused by:
Task state is updated from RUNNING to FAILED on zones/us-central1-f/instances/515175510580775751 due to Spot VM preemption with exit code 50001.
Please suggest. thank you!
t
Does Nextflow actually see the
50001
exit code, i.e. does the exit code of the failed job in the log/execution report show
50001
? Also, any retries are capped by
maxRetries
(process-specific) and
maxErrors
(global) -- make sure those are not limiting. I wonder if your conditioanl expression could be simpler. The nf-core pipelines typically use something like this:
Copy code
errorStrategy = { task.exitStatus in ((130..145) + 104) ? 'retry' : 'finish' }
    maxRetries    = 1
    maxErrors     = '-1'
(Obviously substitute your exit codes, and probably increase
maxRetries
in your case.) I notice you are using
[30,137,247,50001]
, whereas the equivalent in the above construct would probably be
(30 + 137 + 247 + 50001)
. I don't know enough Groovy to say if that makes a difference ... It could also be that this is a GCP-specific problem. In that case, maybe a better place to ask it #infrastructure-google over in the Nextflow Slack ...
p
I suspect that this is one of the issues where there is no
.exitcode
. I've seen
exitStatus
be null in such cases, but at least parts of the code looks like it will try to set it to the maximum value of an int (in java). I would typically recommend reversing the test. I also am somewhat confused by what is returned in the cases where I suspect you don't want to force retries, it looks like it should return the closure? I don't think threre's any default you can fall back on that way.
j
Yes, Nextflow should see that error code from GCP (unlike for AWS where no such specific error code results from spot preemption).
maxRetries
is '1' by default (I think...), so you wouldn't get 3 retries unless you increased that.
l
Thanks for the responses @Thomas Danhorn, @pontus @Jonathan Manning! we have given each process in the workflow an errorStrategy, it could be different for different processes, I generalized in my code below to use
process.errorStrategy
, most of the process.errorStrategy is set to “ignore”. so the code will retry up to 3 times for exit code 30, 137, 247, 50001, then ignore.
errorStrategy { task.exitStatus == null || task.exitStatus in [30,137,247,50001] ? (task.attempt >= 3 ? process.errorStrategy : 'retry') : process.errorStrategy }
We saw retry happens when using an older version of nextflow for exit code 137 etc. but i’m not sure if nextflow actually captures the
50001
exit code. For the specific run that failed due to spot instance: exit-code.txt shows
1
. In the .nextflow.log: `[Task monitor] DEBUG n.c.g.batch.GoogleBatchTaskHandler - [GOOGLE BATCH] Process
duplexPipeline:demux (29)
- last event: description: "Task state is updated from RUNNING to FAILED on zones/us-central1-f/instances/515175510580775751 due to Spot VM preemption with exit code 50001."`
event_time {
seconds: 1736997280
nanos: 929457259
}
type: "FAILED"
task_execution {
exit_code: 50001
}
task_state: FAILED
; exit code: 50001
Jan-16 03:14:44.794 [Task monitor] DEBUG n.processor.TaskPollingMonitor - Task completed > TaskHandler[id: 75; name: duplexPipeline:demux (29); status: COMPLETED; exit: -; error:
....
....
DEBUG nextflow.processor.TaskRun - Unable to dump error of process 'null' -- Cause: java.nio.file.NoSuchFileException: gs://pathtoworkfolderxxxx/.command.err
so maybe nextflow didn’t actually receive 50001?
I didn’t set
maxErrors
, does it has a default value? I will give it a try.
t
I didn’t set
maxErrors
, does it has a default value?
I think the default in most nf-core pipelines is
-1
, meaning infinite, but it does not hurt to make sure by setting it yourself, at least for testing. The exit code messages are confusing, indeed. I have no idea if the
Unable to dump error of process 'null' -- Cause: java.nio.file.NoSuchFileException: gs://pathtoworkfolderxxxx/.command.err
message as caused by the disappearance of the spot instance or is a spearate problem ... But if you look for that specific process in the
.nextflow.log
(there are lines like
[Task monitor] DEBUG n.processor.TaskPollingMonitor - Task completed > TaskHandler[...; name: NFCORE_<PROCESS_NAME> (sample1); status: COMPLETED; exit: <code>;
), it should show you if it died with an exit code and which one. If it was something other than
50001
, maybe add that to the list of "retry" code.
j
To amend my previous response, having checked with colleagues, I think I may have spoken wrongly actually, and 5001 isn't handled all the way to Nextflow, so may not be useful in retry strategies. This may be a bug.
🤔 1
t
So if the process is not exiting with
50001
, the question is then, what is its exit code? If there is none, the
== null
should ideally cover that.
l
exit-code.txt shows
1
. would that been the exit code?
this 1
“most nf-core pipelines is
-1
, meaning infinite,” what does exit code 1 represent? would this cause an issue to retry when exit code =1, that a run could get retry infinitely?
t
Yes, the exit code would be
1
then. Try adding
1
to you conditional expression. The
-1
(infinite) is the setting for
maxErrors
that most pipelines use by default. You could set
maxErrors
to limit the total number of failures, so if you set it to e.g.
3
, a process can either fail 3 times, or three processes can fail once each, or some combination thereof, whereas
maxRetries
applies to each process separately.
maxErrors = -1
means Nextflow will not keep track of the total number of errors (just the per-process errors).
l
Okay. thanks Thomas for the explanation!
👍 1
j
Just to be clear, exit code 1 isn't particularly meaningful, it's just a general “something went wrong” signal. It tells you the program didn’t finish correctly, but doesn’t say exactly why. By contrast, exit code 137 (for example) is more specific—it often means the operating system killed the program because it ran out of memory (OOM). So while exit code 1 is a broad error, exit code 137 points to a memory-related problem. If you're going to put exit code 1 in your list of codes, you may as well just retry unconditionally (up to
maxRetries
).
l
right, yes, thanks Jonathan. good point!
e
@Li Teng Our Nextflow docs indicate Google Batch exit codes like
50001
can be retried but I don't think it was properly implemented, it was flagged again here in this issue: https://github.com/nextflow-io/nextflow/pull/5644. Just a bug that when fixed, will allow you to retry on reclamation.
l
Okay. Thanks Esha for looking into this!