[vsc_ugent profile issue] Hey, we have encountered an interesting issue. We use nextflow (nf-core pipeline, MethylSeq) on the hpc ugent infrastructure and jobs are limited to 72h max. when the pipeline is running for long, there are sometimes jobs that go over the pipeline's 72h. the pipeline is not finished, thus we use

-resume

. However, we encountered duplicates files. this is not problematic most of the time, but in case of a

.collect()

we ended up with duplicates in a .csv file which then caused an error downstream. We suspected that nextflow did not detect the finished job. We think it was caused by few samples having very short reads, therefore the alignment took long. Re-running the pipeline in smaller batches will solve this. my question: is there sth that I missed in catching this error? should not nextflow by default avoid this? What can I do to prevent this again? many thanks for the tips! Tomi

@Nicolas Vannieuwkerke, this smells like a nextflow bug

@Tamás Trombitás do you let all running processes finish before restarting your pipeline? This might cause some issues

@Matthias De Smet version: Nextflow/24.04.2 @Nicolas Vannieuwkerke I waited till all are finished. As I saw, after the job was finished, the initiated processed (separate jobs) were still running with their specified time limits

Yes that is unexpected behaviour, we would like for all processes to stop once the head job shuts down. Could you open a PR to the config repo and reduce the time limit a little bit?

thanks for the answer! I made the PR (reducing process time limit #877) however, I think the issue might still persist(?): especially with larger jobs, I have to queue, since this phenomenon mostly concerns large jobs submitted by the head process, it needs to wait (often a lot) till its started. if at the beginning of the 72h headprocess, a job of 71h59m is queueing 1 day, there is still chance for this "overhang" (provided indeed that the task takes long); and in this scenario the job is being submitted immediately as of the start of the head process. So, if the subprocess is initiated later in the 72h - lets say in the last 2h - there will be a queuing job when the head process has finished already what do you think of this?

The head process should automatically cancel this child process when stopped correctly, which I hope this change will actually do, I'll take a look at the PR

Update: the 71h59m version runs without errors, thanks for the support. however, the behaviour did not change. I submitted the pipeline with 1h time limit (so that the head process surely finished sooner than the alignment). Indeed the head process terminated, but the submitted job (BISMARK_ALIGN) was still running. After that subprocess was finished and I resumed the pipeline, the bismark alignment was performed again (it was not picked up form the cache).

If the head process can’t register the job is finished, then there’s no record in the cache. I think the main issue here is the queue times on the HPC

Meh, that seems like a bug in Nextflow? It would be better that all subprocesses stop if the head job gets cancelled

@Matthias De Smet MethylSeq - with minor in-house modifications (+few modules)

How long does an alignment generally take for one sample?

repends on the read lengths... sometimes we have high fragmentation, therefore the alignment takes very long (+12h). for most OK samples generally not more than few h

so, hot take, IMO there’s no issue with nextflow or the config, it’s just the screwed up wait times on the Ghent HPC 😕 I’d say the only way to go it the give the head job a walltime of 72h, submit and hope your jobs get queued in time 😞

fair enough, thanks for the tips!