Li Teng
01/28/2025, 10:39 PMnextflow 24.10 release
this is in the .nf file for each process:
errorStrategy { task.exitStatus == null || task.exitStatus in [30,137,247,50001] ? (task.attempt >= 3 ? process.errorStrategy : 'retry') : process.errorStrategy }
unfortunately, we don’t see any thing got a retry and a common failure look like this:
Caused by:
Task state is updated from RUNNING to FAILED on zones/us-central1-f/instances/515175510580775751 due to Spot VM preemption with exit code 50001.
Please suggest.
thank you!Thomas Danhorn
01/28/2025, 11:50 PM50001 exit code, i.e. does the exit code of the failed job in the log/execution report show 50001? Also, any retries are capped by maxRetries (process-specific) and maxErrors (global) -- make sure those are not limiting.
I wonder if your conditioanl expression could be simpler. The nf-core pipelines typically use something like this:
errorStrategy = { task.exitStatus in ((130..145) + 104) ? 'retry' : 'finish' }
maxRetries = 1
maxErrors = '-1'
(Obviously substitute your exit codes, and probably increase maxRetries in your case.)
I notice you are using [30,137,247,50001], whereas the equivalent in the above construct would probably be (30 + 137 + 247 + 50001). I don't know enough Groovy to say if that makes a difference ...
It could also be that this is a GCP-specific problem. In that case, maybe a better place to ask it #infrastructure-google over in the Nextflow Slack ...pontus
01/29/2025, 9:03 AM.exitcode. I've seen exitStatus be null in such cases, but at least parts of the code looks like it will try to set it to the maximum value of an int (in java).
I would typically recommend reversing the test.
I also am somewhat confused by what is returned in the cases where I suspect you don't want to force retries, it looks like it should return the closure? I don't think threre's any default you can fall back on that way.Jonathan Manning
01/29/2025, 9:50 AMmaxRetries is '1' by default (I think...), so you wouldn't get 3 retries unless you increased that.Li Teng
01/30/2025, 7:05 AMprocess.errorStrategy, most of the process.errorStrategy is set to “ignore”. so the code will retry up to 3 times for exit code 30, 137, 247, 50001, then ignore.
errorStrategy { task.exitStatus == null || task.exitStatus in [30,137,247,50001] ? (task.attempt >= 3 ? process.errorStrategy : 'retry') : process.errorStrategy }
We saw retry happens when using an older version of nextflow for exit code 137 etc. but i’m not sure if nextflow actually captures the 50001 exit code.
For the specific run that failed due to spot instance:
exit-code.txt shows 1.
In the .nextflow.log:
`[Task monitor] DEBUG n.c.g.batch.GoogleBatchTaskHandler - [GOOGLE BATCH] Process duplexPipeline:demux (29) - last event: description: "Task state is updated from RUNNING to FAILED on zones/us-central1-f/instances/515175510580775751 due to Spot VM preemption with exit code 50001."`
event_time {
seconds: 1736997280
nanos: 929457259
}
type: "FAILED"
task_execution {
exit_code: 50001
}
task_state: FAILED
; exit code: 50001
Jan-16 03:14:44.794 [Task monitor] DEBUG n.processor.TaskPollingMonitor - Task completed > TaskHandler[id: 75; name: duplexPipeline:demux (29); status: COMPLETED; exit: -; error:
....
....
DEBUG nextflow.processor.TaskRun - Unable to dump error of process 'null' -- Cause: java.nio.file.NoSuchFileException: gs://pathtoworkfolderxxxx/.command.err
so maybe nextflow didn’t actually receive 50001?Li Teng
01/30/2025, 7:10 AMmaxErrors , does it has a default value? I will give it a try.Thomas Danhorn
01/30/2025, 6:28 PMI didn’t setI think the default in most nf-core pipelines is, does it has a default value?maxErrors
-1, meaning infinite, but it does not hurt to make sure by setting it yourself, at least for testing.
The exit code messages are confusing, indeed. I have no idea if the Unable to dump error of process 'null' -- Cause: java.nio.file.NoSuchFileException: gs://pathtoworkfolderxxxx/.command.err message as caused by the disappearance of the spot instance or is a spearate problem ...
But if you look for that specific process in the .nextflow.log (there are lines like [Task monitor] DEBUG n.processor.TaskPollingMonitor - Task completed > TaskHandler[...; name: NFCORE_<PROCESS_NAME> (sample1); status: COMPLETED; exit: <code>;), it should show you if it died with an exit code and which one. If it was something other than 50001, maybe add that to the list of "retry" code.Jonathan Manning
01/30/2025, 10:15 PMThomas Danhorn
01/31/2025, 2:13 AM50001, the question is then, what is its exit code? If there is none, the == null should ideally cover that.Li Teng
01/31/2025, 5:54 AM1. would that been the exit code?Li Teng
01/31/2025, 5:59 AM-1, meaning infinite,”
what does exit code 1 represent? would this cause an issue to retry when exit code =1, that a run could get retry infinitely?Thomas Danhorn
01/31/2025, 6:10 AM1 then. Try adding 1 to you conditional expression.
The -1 (infinite) is the setting for maxErrors that most pipelines use by default. You could set maxErrors to limit the total number of failures, so if you set it to e.g. 3, a process can either fail 3 times, or three processes can fail once each, or some combination thereof, whereas maxRetries applies to each process separately. maxErrors = -1 means Nextflow will not keep track of the total number of errors (just the per-process errors).Li Teng
01/31/2025, 6:13 AMJonathan Manning
01/31/2025, 8:45 AMmaxRetries).Li Teng
01/31/2025, 4:22 PMEsha Joshi
02/04/2025, 1:40 AM50001 can be retried but I don't think it was properly implemented, it was flagged again here in this issue: https://github.com/nextflow-io/nextflow/pull/5644. Just a bug that when fixed, will allow you to retry on reclamation.Li Teng
02/04/2025, 1:41 AM