https://nf-co.re logo
Join Slack
Powered by
Hi everyone! Iā€™m running nf-core/mag on UPPMAX and...
# help
p
Hi everyone! Iā€™m running nf-core/mag on UPPMAX and having trouble with the -resume functionality. The pipeline previously reached the assembly (MEGAHIT/metaSPAdes) stage, but the manager process crashed. I am trying to resume with the exact same command and -params-file, but Nextflow is re-running the BOWTIE2_HOST_REMOVAL_ALIGN step from 0%, again even though the .log and output files for that task already exist and were completed in the previous run. I haven't changed the samplesheet, the host FASTA or the params file. Thanks for the help!
Hi again! Update on my 
-resume
issue: Nextflow is completely ignoring my previous Assembly (MEGAHIT/metaSPAdes) and Binning Preparation steps. In the execution console, those processes show as 
[-]
 (not even as 
cached
or 
0%
). However, the Host/PhiX Removal steps are showing as 
[100%] cached
. I have verified that I am in the same working directory and that the 
work/
directory still contains the folders for the assembly tasks.
m
Which cluster and what's your command-line call?
we also have #CPHN84NQ4 too
šŸ‘€ 1
p
I am in Pelle. My original command was: 
nextflow run nf-core/mag -r 3.2.1 -name BankVole_test -profile uppmax --project uppmax2026-1-38 -params-file nf_params.json -resume
To resume, I am using the same command but omitting the 
-name
flag (as Nextflow throws an error that the name 'BankVole_test' already exists). I previously had to resume for other issues and it worked fine.
m
I think you would use 
BankVole_test
after 
-resume
, but to be honest I never use 
-name
. What does 
nextflow log
print? Do you know also if your project got transferred to Gorilla in between runs?
p
My 
.nextflow.log
was overwritten when I last tried to 
-resume
. I manually killed it after seeing it start to repeat the host removal and assembly again. But I did check it before I did 
-resume
and all seemed fine except it crashed because of OOM. I guess Nextflow keeps the whole history of .log files somewhere? But I do not know/haven't checked that yet. But I do have the print of my 
screen
and the assembly .log files for both 
SPAdes
and 
MEGAHIT
if you wanna see that?
I could also try to run it with 
-resume
Bank_Vole
?
m
The old log should still be in the directory. The 
.nextflow.log
gets moved to 
.nextflow.log.1
when a new run starts. The last 10 runs are saved. I meant the command though. If it's only showing the last run, then you didn't run your workflow from the same directory and is why it's restarting
p
Hmmm, let me see that!
ok, this is what the 
nextflow.log
you asked looks like: [pilarlp7@pelle2 Pilar]$ tail -f .nextflow.log.2 Mar-25 061308.141 [Task submitter] DEBUG n.processor.TaskPollingMonitor - %% executor slurm > tasks in the submission queue: 1 -- tasks to be submitted are shown below ~> TaskHandler[jobId: null; id: 15; name: NFCORE_MAGšŸ”QUAST (MEGAHIT-sample1985); status: NEW; exit: -; error: -; workDir: /crex/proj/microbiome/BankVole/Metagenome/Pilar/work/a2/9124bc8a5b0e7926341ed9b18addb8 started: -; exited: -; ] Mar-25 061308.199 [main] DEBUG nextflow.trace.ReportObserver - Workflow completed -- rendering execution report Mar-25 061308.602 [main] DEBUG nextflow.trace.TimelineObserver - Workflow completed -- rendering execution timeline Mar-25 061308.641 [main] DEBUG nextflow.trace.TraceFileObserver - Workflow completed -- saving trace file Mar-25 061308.678 [main] DEBUG nextflow.cache.CacheDB - Closing CacheDB done Mar-25 061308.807 [main] INFO org.pf4j.AbstractPluginManager - Stop plugin 'nf-schema@2.5.1' Mar-25 061308.807 [main] DEBUG nextflow.plugin.BasePlugin - Plugin stopped nf-schema Mar-25 061308.807 [main] DEBUG nextflow.util.ThreadPoolManager - Thread pool 'FileTransfer' shutdown completed (hard=false) Mar-25 061308.808 [main] DEBUG nextflow.script.ScriptRunner - > Execution complete -- Goodbye
m
nextflow
has a 
log
subcommand. I was asking for the output of that.
p
haha sorry
[pilarlp7@pelle2 Pilar]$ nextflow log TIMESTAMP DURATION RUN NAME STATUS REVISION ID SESSION ID COMMAND 2026-03-20 151636 7h 33m 29s BankVole_test ERR 0c370baf73 5af62d15-5d65-4264-b619-b58f3a5f9945 nextflow run nf-core/mag -r 5.4.1 -name BankVole_test -profile uppmax --project uppmax2026-1-38 -resume -params-file nf_params.json 2026-03-22 213104 5h 59m 33s sick_fermat ERR 0c370baf73 5af62d15-5d65-4264-b619-b58f3a5f9945 nextflow run nf-core/mag -r 5.4.1 -resume -profile uppmax --project uppmax2026-1-38 -params-file nf_params.json 2026-03-23 122215 - backstabbing_boltzmann - 0c370baf73 5af62d15-5d65-4264-b619-b58f3a5f9945 nextflow run nf-core/mag -r 5.4.1 -profile uppmax --project uppmax2026-1-38 -resume -params-file nf_params.json 2026-03-23 153300 1d 14h 40m 9s insane_pauling ERR 0c370baf73 5af62d15-5d65-4264-b619-b58f3a5f9945 nextflow run nf-core/mag -r 5.4.1 -profile uppmax --project uppmax2026-1-38 -resume -params-file nf_params.json 2026-03-25 083730 - crazy_knuth - 0c370baf73 5af62d15-5d65-4264-b619-b58f3a5f9945 nextflow run nf-core/mag -r 5.4.1 -profile uppmax --project uppmax2026-1-38 -resume -params-file nf_params.json 2026-03-25 085728 - dreamy_pauling - 0c370baf73 5af62d15-5d65-4264-b619-b58f3a5f9945 nextflow run nf-core/mag -r 5.4.1 -profile uppmax --project uppmax2026-1-38 -params-file nf_params.json -resume
(I actually tried that command earlier today and didn't work but I was probably on the wrong place...)
m
If you do 
nextflow clean -f -after BankVole_test
That should clean up the runs after the 
BankVole_test
. Then try with 
-resume
again
p
ok, "clean BankVole" sounds scary though šŸ« šŸ˜…
m
You can try with 
-n
instead of 
-f
first so you can see what will get removed
p
I will use the command line to -resume that worked before (no -name or BankVole after -resume)
m
Also use 
-resume BankVole_test
just to make sure it starts from there
Oh wait. It needs the ID, not name
I think it should be 
-resume 5af62d15-5d65-4264-b619-b58f3a5f9945
p
aha
that sounds good!
thanks so much! let's see!
I checked with `-n`first and I have confirmed that it would remove folders containing the output of both SPAdes and MEGAHIT
Instead of cleaning, is there a way to force Nextflow to resume from the assembly session? I could clean the two "bad" sessions maybe and then resume?
m
Use the session id corresponding to the run with the successful assemblies
p
ok!
m
Did it work?
p
no šŸ˜‚ but tbh I did run 
-resume insane_pauling
which is the run name where the assemblies were done because the session ID is the same for all of them... I was planning trying as well with ID if run name didn't work but I decided I will use the nf-core/mag option of starting with the assemblies, since I already got them and I am consuming a lot of the project time šŸ«  But I won't rest in peace until I understand why nextflow keeps on starting a new job for something that is been already done without complains šŸ§
m
šŸ‘€ 2
p
Thank you so much!
2 Views
Open in Slack
PreviousNext
Powered by