Hi everyone. I'm not sure if this might be the right channel for this, but I am having an issue with resuming a scnanoseq pipeline run. When I initially tried to resume the pipeline (using the same command as my initial run but with

-resume

added), the pipeline failed due to inability to acquire lock. I then manually deleted the LOCK file and resumed the pipeline again, which worked this time, but instead of resuming from the last step that failed, it is re-doing some of the earlier step as it was unable to find the cache for those? The initial run of the pipeline failed after 36 hours so I would ideally not want to re-do the computation again if I can resume just from the last failed step. Is there more information available on how to resume a scnanoseq pipeline run? Thanks!

Hi Andrian,

-resume

should be all that's needed. What was the reason for failure on the previous run? The reason I ask is normally the LOCK files are cleaned up when the nextflow runner exits/completed, but if the LOCK file still existed after the pipeline runner had exited its possible the runner was unable to exit gracefully and unable to cache the state of jobs correctly, which can cause issues with being able to assess the state of the pipeline

Hi Austyn. Thanks for getting back to me! I am running the pipeline on a cluster and we unfortunately have a max job run limit time of 36 hours, so the job got killed most likely as it was in the middle of running a process. I actually had another pipeline running (to process another FASTQ file) and it seems that I was able to resume that job no problem, but not the one I am reporting on above.

And I agree, it does look like it was not resuming the correct session. You can also pass in a session id/run name as an option to the

-resume

to ensure nextflow resumes the correct run

Yes. I did that on line 6, but the run failed due to 'missing required parameter'. I ended up having to include all the parameters again and then appending the ID at the end (see line 7). I feel that this may be a bug since the example of the resume command on the nextflow documentation does not require re-specifying the parameter at all - just the ID as you mentioned.

Hi Andrian - one note from reading this (given the limitations with the 35 hr limit), is that you may want to limit quantification to using the

oarfish

pathway. IsoQuant is known for taking longer, so if you are limited with time, just an FYI (if you can get around this, then no problem). I only add this as a side note - I think Austyn is correct regarding the resume issues.

Hi Lara. Thanks for the feedback. We would ideally want to compare the short read data against the long read data so I'm not sure

oarfish

pathway is the optimal option for us. From reading the documentation, I am currently using fastq splitting in order to make sure that each step should not take too long to run and from the job duration report of another sample, it looks like the isoquant runtime is relatively not too long.

Ok if IsoQuant is not taking too long with your data (it all depends on depth), then great. Yes - only caveat with oarfish is that it provides transcript-level quantification only (not transcript and gene-level). You would need to do additional processing with oarfish to summarize it down to gene-level. that being said, it did remind me to touch base with Mike Love regarding using tximport with oarfish outputs (to summarize it to gene-level). I am hoping that this is a path in the future for oarfish. Let us know if you solve the resume issue in the meantime. This is not an issue that is reproduced on our end, but happy to provide further advice if needed.

I've managed to solve the issue for the resume - adding the correct run ID hash seems to be the solution. I thought previously that it should automatically calculate the correct hash based on the values of the parameter config I supplied, but that does not seem to be the case.

Good to know. That is odd though, we have never encountered this problem. Let us know if it continues to happen (if so we would need to find a way to replicate or maybe ask others here in the nextflow/nf-core community on the other more general channels).

Just wanted to update that using

-resume

with run-ID has been working very nicely with 3 separate scnanoseq workflows running concurrently. I saw that the usage documentation mentioned that this is an option, but can I suggest setting this as the default recommended method?

Hi Andrian - just need clarification on your latest comment. What specifically do you mean setting this as default? You mean setting

-resume

with run-ID as the default method?

Yes. My understanding from reading the current documentation on

-resume

is that we just need to add

-resume

to the existing nextflow invocation to resume that pipeline, with the next paragraph suggesting that we could also do

-resume [run-name]

. However, I think it would be better to recommend that the way to use

-resume

is to use it by adding

-resume [run-ID|run-name]

to the existing nextflow invocation instead of this being an optional thing.

OK sounds good - we can make a change in the documentation section. This is still for me the one odd issue since

-resume

for us has typically worked. But yes @Austyn Trull and I are happy to make a change in the docs.

I am a beginner in nextflow, so this is just a guess from my end, but according to the nextflow documentation here, the

-resume

will resume from previous runs by default. Since I had multiple scnanoseq runs (one for each sample), I think this causes nextflow to get confused and it picked the incorrect run to resume from. This could just be an issue with the way I am running scnanoseq though, since I understand that we can technically run multiple samples in a single run.

yes - we always run with multiple samples in a single run. So this may very well be it.

Yes, I am running the different pipeline runs in the same working directory (though different output directory) so that is probably the cause! I am fairly new to nextflow, so i didn't realise that we need separate working directory for different pipeline runs? I thought that the working directory is meant to be shared in general?

Well - it could work but it is just something we can’t guaranteed since we haven’t heavily tested it in that way (since we typically provide multiple samples as a single run). But that also certainly explains the

-resume

issue you have encountered. If you have multiple runs, it makes sense why you would need to provide the specific ID.

I see. I think I prefer having separate runs for each sample as I can clean a completed job and start another run (and resume from existing failed job) given the space 'limitation' that I have.

No problem - I totally get it. Just wanted to say we haven’t tested in the way you are running, but if you are not encountering any obvious issues (beyond the resume caveat), then no problem.

So far, I've been able to successfully resume the workloads, but every one of them seems to be crashing at different state due to running out of disk space and/or resource limitation

Can you share the custom config that you are using so that Austyn and I can check the resource request? We may be able to provide some advice (regarding the disk space - not much we can do at the moment until other methods make it to a next release)

I'll also ask, are you running the pipeline locally or on a shared compute environment?

On the Cambridge HPC environment. I was previously running the netxflow pipeline in a single job (i.e. ask for a 30 core, 200GB job and run all the pipeline within that job), though I have now pivoted to running the nextflow pipeline with the cambridge (slurm) config where each step in pipeline is now submitted as individual job.

correct - that is exactly what we recommend so that multiple steps can be executed in parallel through Nextflow’s executor mode.

This is primarily due to the 12 hour SLURM job limit. The queue can be lengthy at times, in which case having a single job would be more preferable.

One point to note, is that you can then also request resources for specific processes - (in terms of memory etc…). We are happy to review your resource request if you would like feedback. And regarding, my only real concern with with time (for specific steps) is IsoQuant, but if you already got through this in the past then it should be fine.

Yes. IsoQuant wasn't the biggest issue for me - it was BLAZE and minimap2, though BLAZE is now manageable after turning off demultiplexing

Does your institution not have a scratch space? Where disk storage is higher with the goal of deleting intermediate files once the analysis is complete?

Unfortunately there is no such space in the Cambridge HPC. The scratch space is basically the 'group' drive which we need to purchase capacity for (in duration of minimum of 1 year). As such, it would be ideal to not have to purchase a lot of extra 'scratch' space for the purpose of analysing this data.

Understood - I wish we could get to other improvements sooner, but unfortunately we won’t be able to prior to 2026 Q1. But what we already have there is a good starting point. The only thing I can think of, but may not be worth implementing here, is the nf-boost plug in: https://github.com/bentsherman/nf-boost/tree/main This deletes temporary files in the work dir as the pipelines gets executed but I have never actually tested it. Also a big tradeoff here is that you would not be able to resume the runs since the temp. files are deleted. So you would need to know for certain that the resources you are requesting etc. are sufficient. Thought it is worth mentioning anyway (and again, not tested by us).

I did consider it, but I think the resume-ability is a helpful feature for now while are testing and tweaking the pipeline - for example a pipeline which just recently completed failed at the Seurat step due to OOM error...

Yes correct - you would need to focus first on a single sample (likely the sample that you have highest depth) first to narrow down resource requests. Once you feel comfortable, maybe this is worth. But certainly not until then