Hello, I have a general question. Im using the MAG pipeline and Im having issues with the

-resume

. Especially today because my cluster was in maintenance for a couple days and now its starting from the beginning. Is this normal? Is there a way to solve it? Why sometimes it doesnt cache all the things that have already been done? Also if the main jobs is cancel but some of the submited jobs are still running and finish correctly, are those be cache in the next main job?

To understand a bit better, did you run it with

--resume

when you submitted it originally? (And if you look in the output dir are there partially completed files and/or info in the

.nextflow.log?

I did run it with

-resume

originally and yes, the output is there, the work directory is there and even when I do

nextflow log

the runs show up but its not chaching anything...

And just to double check, you did have steps finish?

I had some finish and some of them failed after the parent job failed because some of them were already submited. Abnd if that was the issue the first parts of the pipeline would be caching and only the end would need to be redone but its redoing everything from the start.

Yea that is weird, it should be finding those I agree.

can

-resume run_name

be use to force the caching of the last run that worked? or some other id or file to force the caching?

If you run

nextflow log

it should show you which runs it is aware of. I think you should be able to put in the session id or the run name. ie if you have a run name of "big_plank" something like this:

nextflow run nf-core/mag -resume big_plank ....

that's already what I did and it still didnt cached anything.

nextflow log 
TIMESTAMP     	DURATION  	RUN NAME     	STATUS	REVISION ID	SESSION ID             	COMMAND                                                                                                                                                                                                                                                                                                                          
2026-03-30 17:29:07	20h 20m 13s	friendly_tuckerman	ERR  	0c370baf73 	5c6ea3f0-f979-408f-9a37-f6730d2f5b91	nextflow run nf-core/mag -profile ifb_core -r 5.4.1 -c mod.config --input /shared/projects/mags_margo/FL/data/samplesheetFL.csv --outdir /shared/projects/mags_margo/FL/output/nf_core_mag --multiqc_title FL_samples --fastp_trim_polyg --gtdb_db /shared/bank/gtdbtk/release226/ --skip_spades --run_checkm2 --checkm2_db /shared/bank/checkM2/version_3/CheckM2_database/uniref100.KO.1.dmnd --refine_bins_dastool --postbinning_input both -resume --igenomes_base false --checkm_db /shared/ifbstor1/bank/checkm/2015-01-16/                                                             
2026-03-31 19:41:30	56m    	ridiculous_gates 	ERR  	0c370baf73 	5c6ea3f0-f979-408f-9a37-f6730d2f5b91	nextflow run nf-core/mag -profile ifb_core -r 5.4.1 -resume friendly_tuckerman -c /shared/projects/mags_margo/FL/nf-core-mag/mod.config --input /shared/projects/mags_margo/FL/data/samplesheetFL.csv --outdir /shared/projects/mags_margo/FL/output/nf_core_mag --multiqc_title FL_samples --fastp_trim_polyg --gtdb_db /shared/bank/gtdbtk/release226/ --skip_spades --run_checkm2 --checkm2_db /shared/bank/checkM2/version_3/CheckM2_database/uniref100.KO.1.dmnd --refine_bins_dastool --postbinning_input both --igenomes_base false --checkm_db /shared/ifbstor1/bank/checkm/2015-01-16/                              
2026-04-03 16:47:39	21.5s   	big_hypatia    	OK  	0c370baf73 	5c6ea3f0-f979-408f-9a37-f6730d2f5b91	nextflow run nf-core/mag -profile ifb_core -r 5.4.1 -w /shared/projects/mags_margo/FL/nf-core-mag/work -resume ridiculous_gates -c /shared/projects/mags_margo/FL/nf-core-mag/mod.config --input /shared/projects/mags_margo/FL/data/samplesheetFL.csv --outdir /shared/projects/mags_margo/FL/output/nf_core_mag --multiqc_title FL_samples --fastp_trim_polyg --gtdb_db /shared/bank/gtdbtk/release226/ --skip_spades --run_checkm2 --checkm2_db /shared/bank/checkM2/version_3/CheckM2_database/uniref100.KO.1.dmnd --refine_bins_dastool --postbinning_input both --igenomes_base false --checkm_db /shared/ifbstor1/bank/checkm/2015-01-16/ -preview
2026-04-03 16:50:29	-     	special_sinoussi 	-   	0c370baf73 	5c6ea3f0-f979-408f-9a37-f6730d2f5b91	nextflow run nf-core/mag -profile ifb_core -r 5.4.1 -w /shared/projects/mags_margo/FL/nf-core-mag/work -resume ridiculous_gates -c /shared/projects/mags_margo/FL/nf-core-mag/mod.config --input /shared/projects/mags_margo/FL/data/samplesheetFL.csv --outdir /shared/projects/mags_margo/FL/output/nf_core_mag --multiqc_title FL_samples --fastp_trim_polyg --gtdb_db /shared/bank/gtdbtk/release226/ --skip_spades --run_checkm2 --checkm2_db /shared/bank/checkM2/version_3/CheckM2_database/uniref100.KO.1.dmnd --refine_bins_dastool --postbinning_input both --igenomes_base false --checkm_db /shared/ifbstor1/bank/checkm/2015-01-16/

and my code is

#!/bin/bash
#
#SBATCH -o logs/MAG.log_%J.out
#SBATCH -e logs/MAG.log_%J.err
#SBATCH --mail-type END
#SBATCH --mail-user <mailto:carla.pereira-garcia@univ-littoral.fr|carla.pereira-garcia@univ-littoral.fr>
#SBATCH -p long
#

cd /shared/projects/mags_margo/FL/nf-core-mag

module purge
module load nextflow
module load singularity
module load openjdk

nextflow run nf-core/mag \
	-profile ifb_core \
	-r 5.4.1 \
	-w /shared/projects/mags_margo/FL/nf-core-mag/work \
	-resume ridiculous_gates \
	-c /shared/projects/mags_margo/FL/nf-core-mag/mod.config \
	--input /shared/projects/mags_margo/FL/data/samplesheetFL.csv \
	--outdir /shared/projects/mags_margo/FL/output/nf_core_mag \
	--multiqc_title FL_samples \
	--fastp_trim_polyg \
	--gtdb_db /shared/bank/gtdbtk/release226/ \
	--skip_spades \
	--run_checkm2 \
	--checkm2_db /shared/bank/checkM2/version_3/CheckM2_database/uniref100.KO.1.dmnd \
	--refine_bins_dastool \
	--postbinning_input both \
	--igenomes_base false \
	--checkm_db /shared/ifbstor1/bank/checkm/2015-01-16/

I haven't tried this myself, but it looks like there is a "dump-hashes" command that you might be able to use to figure out what is actually being saved/why things are being restarted. Link

dont undestand really what this does but it looks like you have to added to the run and I didnt do that in the beginning so I dont think its going to help...

Yes, the

-dump-hashes

option is mostly helpful if you used it in the original run and any resumed ones. But you can test with a small sample if you can reproduce the problem and how it is affected e.g. by using different parameters. It is unfortunately not uncommon that certain processes will be rerun (instead of using the cache), mostly because of imperfect pipeline implementation that subtly changes what exactly is passed to a process. But if absolutely nothing is used from cache this may be a more fundamental problem. You are running the same pipeline command (with

-resume

) in the same directory when you are resuming, right?

Yes, of course. Same pipeline, same version, same directory, same script, same everything. The only thing that has change is that the cluster was under maintenance before and after

Here are all the things that go into the hash that determines whether two attempts to run a process are "the same", i.e. a cached attempt can be reused. Are you using a container engine (Docker, Apptainer, etc.), or Conda?