Hi, I have a problem running metadenovo on dardel. It seems to run fine until the megahit step (decided for megahit as it saves resources no?). I run sediment RNA samples (n=33) and also water RNA (n=15) and both stop at the same spot at megahit. It runs for a while (1-2 h) and then the pipeline complete with errors with the following message: Caused by: "Process

NFCORE_METATDENOVO:METATDENOVO:MEGAHIT_INTERLEAVED (megahit_assembly)

terminated for an unknown reason -- Likely it has been terminated by the external system" I was thinking maybe it has something to do with the amount of resources used? So i tried first without a config file, and then afterwards with a config file using those parameters: params { max_memory = 50.GB max_cpus = 20 max_time = 7.d } I am not familiar with dardel that much (already had struggle with 16S analysis before) and was wondering if someone might have an idea how I can fix this? Could it be that I need to adjust the parameters for resource use? Or am I completely wrong and the error is something else? Or should I better ask my question in the helpdesk-hpc-sweden channel? Thanks for your help!

Resources, more specifically memory, is likely the problem. I'd try to give it 128 GiB to start with. Since you "pay" 1 cpu core per GiB memory you will very soon run into problems with running times -- your jobs won't start. The way around this is to be more conservative with the time than we had to at Rackham. I'd hence try a config with this:

process {
    withName: MEGAHIT_INTERLEAVED {
        cpus   = { 64 }
        memory = { 128.GB }
        time   = { 3.d }
    }
}

(I'm also attaching a config I recently used. For this I had to ask for even more memory.) Note that you can't have

params

in configs any more (see warning note under https://nf-co.re/metatdenovo/1.2.0/#usage-1) and that the

max_*

params have disappeared.

thank you! I will give it a try, I was also trying to solve the problem within the helpdesk-hpc-sweden but we did not succeed yet.

Did you try my example above, i.e. 128 GiB RAM? You probably need the memory and with memory comes cpus, so the only thing you can fiddle with is the duration. In my experience, getting a

(AssocGrpCPURunMinutesLimit)

is not a showstopper unless you're way above your allocated resources, i.e. your job might start tonight, during a weekend or so. Which queue jobs end up in -- memory or long -- is difficult to predict as are the consequences. If nothing works, you can try

--bbnorm

which will downsample your raw data before assembly so that less memory is required. (The full sized libraries are used for quantification.) A more longterm solution is to apply for a larger compute project. "Small" projects are now up to 20,000 core hours. Recently we got that after my arguing for the need to have access to a lot of memory. Get back to me if you want help with that.

Hi, yea I tried both of your suggestions but I am already far over my core-hours. I wrote the support team now, they probably will tell me to increase my core-hours as well. I will try to run it again and wait for the support reply, otherwise I might apply for more hours. And how would I use --bbnorm to downsample ?

--bbnorm

is a parameter to the pipeline that uses BBNorm (https://archive.jgi.doe.gov/data-and-tools/software-tools/bbtools/bb-tools-user-guide/bbnorm-guide/) to downsample raw sequence reads to reduce sequencing depth of very abundant sequences. You just add that to the pipeline run and it will start by doing this for all samples before continuing with assembly. Theoretically, this could improve assembly quality not just help you to get it through. In practice, we have seen that you get fewer ORFs and smaller proportions mapping back to the assembly. I believe this is mostly quite rare transcripts though.

I asked for more core hours but havent heard back from them zet, while waiting I tried --bbnorm which starts but fails again witthe the following reason; Caused by: Process

NFCORE_METATDENOVO:METATDENOVO:BBMAP_BBNORM (all_samples)

terminated for an unknown reason -- Likely it has been terminated by the external system

This is likely memory (again!). With BBNorm it's a bit more complicated as it seems it asks for more memory than we give it. I have tried to fix this by giving it only 95% of the memory we actually have, but it still fails. It could also be that it actually needs more memory itself. Difficult to say. If you start by giving it, say, 120 GiB, and resume, I'll update the code to reduce the memory the process sees to 80% so you can try that if it still fails.

I will give it a try 🙂

Thanks! I just started a run with the same modification in #CE9MS66BS setting the memory to 80%. I'm giving the process 120 GiB so I'm doing both at the same time -- never do that if you want to find out what the problem is! 😉

I will give it a try, thank you!

@Ben Pontiller: Also for you!

My problem now is that I have not the latest version of Nextflow (or a too old one) when I am working on Dardel. I use module load PDC nextflow apptainer python/3.12.3 to load nextflow. When i try the self-update it says permission denied. Which command should I use to update the version?

Oh! Each nf-core pipeline specifies its minimum Nextflow version. Usually quite a recent version, without perhaps really needing it but it makes developers lives easy (at least mine). Computing centers might have difficulties to keep up to date, so I use my own Nextflow. Unload nextflow and install it:

mkdir ~/bin
cd ~/bin
curl -s <https://get.nextflow.io> | bash

The above might complain about your Java version. I think a

module load java/OpenJDK_17+35

might help in that case. If not, get back to me. Also, make sure

~/bin

is in your

PATH

environment variable (check with:

env | grep PATH

and inspect).

great thanks! 🙂 lets try that

When you get that kind of error, try

module spider java/OpenJDK_17+35

. This time it tells you that you need to load

bioinfo-tools

first.

oh damn, totally forgot to load that haha 😄

Yes, maybe the easiest way to check if if you do

which nextflow

. If that says

which: no nextflow in ...

(assuming you have installed it) then you need to add it:

export PATH=~/bin:$PATH

. Add this line also to your

~/.bash_profile

(or

.profile

if the former doesn't exist).

it shows me the path with ../../bin/nextflow after I asked which nextflow (unloaded nextflow module)

Perfect, you're ready to go!

If I want to run nextflow now with my downloaded version, how should I do it in a screen session? I usually do a module load for nextflow, but if I unload the module now, do I have to use the PATH to it?

In a running screen session, you need to unload the module (if loaded) and update the

PATH

variable. If you've updated

.bash_profile

and don't mind starting a new session, "source"

.bash_profile

(

source .bash_profile

or, old-fashioned,

. .bash_profile

) and start a new screen session.

Good it started, lets see what happens 🙂

Nice, keep me posted!

I am now at the step of BBNORM but it is in queue for a long partition but so far has priority (Nodelist). Not sure when it will star though

How much time did you ask for? I think 1-2 days should suffice.

I asked 120 gb for 3days , should I reduce it?

Maybe. It could be that you ended up in the "long" queue because of this. If that's worse than the other queues, I don't know. If you cancel, update and resume, you'll of course lose your place in the queue... It depends a bit on whether you're relying on bonus time or if you still have time. Getting bonus jobs through the queue is more difficult the longer they are.

I think for this month I rely on bonus time, already above the computing time I have, Still was not able to get info if I can increase the time.

OK, that speaks for cancelling, asking for shorter time and resuming. But I'm still not sure if that's the best option...

I will try with that one project. I still have another one ( was running before you mentioned your adapted pipeline) also in long queue for forever for BBNORM, so I guess I need to cancel as nothing goes forward otherwise 😄

Sorry for tricking you into restarting... 🙂

Was worth a try! I guess I need to wait now and hope I get a soon reply from the support team for more resources 🙂

In my experience, "AssocGrpCPURunMinutesLimit" does not mean the job won't ever start. So, keep the session running and hope the jobs start.

Yes I will 🙂

Just let the jobs remain in the queue, i.e. let nextflow run. Check your allocation with

projinfo -u your_user

it did fail for both projects but this time the command error is this: INFO: Mounting image with FUSE. fuse: unknown option(s): `-o uid=3223195,gid=3223195' WARNING: squashfuse mount failed, falling back to extraction: encountered error while trying to mount image "/cfs/klemming/scratch/l/lause/community-cr-prod.seqera.io-docker-registry-v2-blobs-sha256-5a-5aae5977ff9de3e01ff962dc495bfa23f4304c676446b5fdf2de5c7edfa2dc4e-data.img" with FUSE at /tmp/rootfs-2329280659/root: exit status 1

I think I get that all the time too. I think it has something to do with what Dardel supports and not, but doesn't actually affect your run. Can you paste the full error message you get?

Aug-18 211427.889 [Task monitor] DEBUG n.processor.TaskPollingMonitor - Task completed > TaskHandler[jobId: 11742486; id: 48; name: NFCORE_METATDENOVOMETATDENOVOBBMAP_BBNORM (all_samples); status: COMPLETED; exit: -; error: -; workDir: /cfs/klemming/scratch/l/lause/RNA/snpseq01320/work/22/a4cd228ff93e9b6d106211ddc088a2 started: 1755540185855; exited: -; ] Aug-18 211428.563 [TaskFinalizer-6] DEBUG nextflow.processor.TaskProcessor - Handling unexpected condition for task: name=NFCORE_METATDENOVOMETATDENOVOBBMAP_BBNORM (all_samples); work-dir=/cfs/klemming/scratch/l/lause/RNA/snpseq01320/work/22/a4cd228ff93e9b6d106211ddc088a2 error [nextflow.exception.ProcessFailedException]: Process

NFCORE_METATDENOVO:METATDENOVO:BBMAP_BBNORM (all_samples)

terminated for an unknown reason -- Likely it has been terminated by the external system Aug-18 211429.078 [TaskFinalizer-6] ERROR nextflow.processor.TaskProcessor - Error executing process > 'NFCORE_METATDENOVOMETATDENOVOBBMAP_BBNORM (all_samples)' Caused by: Process

NFCORE_METATDENOVO:METATDENOVO:BBMAP_BBNORM (all_samples)

terminated for an unknown reason -- Likely it has been terminated by the external system Command executed: bbnorm.sh \ in=YC-4250-BIO74water_S53_L003_R1_001.fastq.gz.processed.fastq.gz,YC-4250-BIO78water_S54_L003_R1_001.fastq.gz.processed.fastq.gz,YC-4250-BIO89water_S63_L003_R1_001.fastq.gz.processed.fastq.gz,YC-4250-BIO120water_S66_L003_R1_001.fastq.gz.processed.fastq.gz,YC-4250-BIO116water_S64_L003_R1_001.fastq.gz.processed.fastq.gz,YC-4250-BIO85water_S60_L003_R1_001.fastq.gz.processed.fastq.gz,YC-4250-BIO88water_S62_L003_R1_001.fastq.gz.processed.fastq.gz,YC-4250-BIO73blank_S52_L003_R1_001.fastq.gz.processed.fastq.gz,YC-4250-BIO80water_S56_L003_R1_001.fastq.gz.processed.fastq.gz,YC-4250-BIO82water_S57_L003_R1_001.fastq.gz.processed.fastq.gz,YC-4250-BIO83water_S58_L003_R1_001.fastq.gz.processed.fastq.gz,YC-4250-BIO84water_S59_L003_R1_001.fastq.gz.processed.fastq.gz,YC-4250-BIO79water_S55_L003_R1_001.fastq.gz.processed.fastq.gz,YC-4250-BIO86water_S61_L003_R1_001.fastq.gz.processed.fastq.gz,YC-4250-BIO117water_S65_L003_R1_001.fastq.gz.processed.fastq.gz \ out=all_samples.fastq.gz \ target=100 min=5 \ threads=64 \ -Xmx96g \ &> all_samples.bbnorm.log cat <<-END_VERSIONS > versions.yml "NFCORE_METATDENOVOMETATDENOVOBBMAP_BBNORM": bbmap: $(bbversion.sh | grep -v "Duplicate cpuset") END_VERSIONS Command exit status: - Command output: (empty) Command error: INFO: Mounting image with FUSE. fuse: unknown option(s): `-o uid=3223195,gid=3223195' WARNING: squashfuse mount failed, falling back to extraction: encountered error while trying to mount image "/cfs/klemming/scratch/l/lause/community-cr-prod.seqera.io-docker-registry-v2-blobs-sha256-5a-5aae5977ff9de3e01ff962dc495bfa23f4304c676446b5fdf2de5c7edfa2dc4e-data.img" with FUSE at /tmp/rootfs-1806610916/root: exit status 1 INFO: Converting SIF file to temporary sandbox... WARNING: underlay of /etc/localtime required more than 50 (74) bind mounts

I think this is the key:

terminated for an unknown reason -- Likely it has been terminated by the external system

. I just responded to a question @Ben Pontiller has with BBNORM (unfortunately in a DM -- would be more useful here 😉 ): I think the

bgzip

part is a distraction. Instead, I think the key is in this part:

For better accuracy, use the 'prefilter' flag; run on a node with more memory; quality-trim or error-correct reads; or increase the values of the minprob flag to reduce spurious kmers.  In practice you should still get good normalization results even with loads over 90%, but the histogram and statistics will be off.

Can you ask for more memory? If so, try to double. If not, add the below to your

nextflow.config

(or whatever name you use):

process {
    withName: BBNORM {
        ext.args = { [
            "target=${params.bbnorm_target}",
            "min=${params.bbnorm_min}",
            "prefilter"
        ].join(' ').trim() }
        ... whatever options you already have
    }
}

If you, @Laura Seidel, are running the

erikrikarddaniel/metatdenovo -r lower-bbnorm-memory

version, try the above or -- again! -- more memory. As I think I mentioned, I got a number of very large BBNorm processes through the #CE9MS66BS pipeline with the below config on Dardel:

withName: BBMAP_BBNORM {
        memory = 120.GB
        cpus = 48
        time = 3.d
    }

Thanks! I will check that out. I had to restart it now on my own project storage space, as I got more storage and more computing. I will see if it will get through. I cant change the config while it is running, that would probably crash something, wouldnt it?

It won't crash anything, but it won't take effect either.

yes I used the scratch area since now but I am almost struggle for 3 weeks with this now and didnt get any further so I have now more storage and transferred everything, to be on the safe side

The reduction in memory for

BBNORM

is now in the

dev

branch of the official pipeline, i.e.

nextflow run nf-core/metadenovo -r dev -latest

. What does the

.command.log

in the work directory say? (The last lines usually contain a message like "oom [something]" or at least a return code (143 indicating out of memory).) Also check

.exitcode

.

stupid question. i do know how to check the .nextflow.log but when I try the same with .command.log or the others, it says it cannot find it. How can I check them?:)

NOT a stupid question (there aren't any here #C043FMKUNLB 😉 ). These files are in the work directory for the process. Normally when a process fails you can see the path to this at the end of the error message. You can also

grep

.nextflow.log

to find it. An example from one of my recent runs:

$ grep "EUKULELE_SEARCH.*workDir" .nextflow.log
aug.-22 07:43:24.557 [Task monitor] DEBUG n.processor.TaskPollingMonitor - Task completed > TaskHandler[id: 26; name: NFCORE_METATDENOVO:METATDENOVO:SUB_EUKULELE:EUKULELE_SEARCH (megahit.prodigal); status: COMPLETED; exit: 0; error: -; workDir: /data/eukulele-test/work/48/2669cccbd7969bce5d2ff982b14eb5]

The path after

workDir

is what you're looking for.

cd

into that and you'll find the files I mentioned.

the .command.log shows the following: Unloading the cpe module is insufficient to restore the system defaults. Please run 'source /opt/cray/pe/cpe/24.11/restore_lmod_system_defaults.[csh|sh]'. Sourcing the above script is not necessary if swapping the cpe module with a different version. The following have been reloaded with a version change: 1) gcc-native/12.3 => gcc-native/13.2 INFO: Environment variable SINGULARITYENV_TMPDIR is set, but APPTAINERENV_TMPDIR is preferred INFO: Environment variable SINGULARITYENV_NXF_TASK_WORKDIR is set, but APPTAINERENV_NXF_TASK_WORKDIR is preferred INFO: Environment variable SINGULARITYENV_NXF_DEBUG is set, but APPTAINERENV_NXF_DEBUG is preferred INFO: fuse2fs not found, will not be able to mount EXT3 filesystems INFO: gocryptfs not found, will not be able to use gocryptfs slurmstepd: error: Detected 1 oom_kill event in StepId=11863416.batch. Some of the step tasks have been OOM Killed. and thats the start of the .command.run: #SBATCH -J nf-NFCORE_METATDENOVO_METATDENOVO_BBMAP_BBNORM_(all_samples) #SBATCH -o /cfs/klemming/scratch/l/lause/RNA/snpseq01320/work/90/a5497cea61f34b022dd75e1dc80778/.command.log #SBATCH --no-requeue #SBATCH --signal B:USR2@30 #SBATCH -c 64 #SBATCH -t 240000 #SBATCH --mem 56320M #SBATCH -A naiss2024-22-1152 -p shared NXF_CHDIR=/cfs/klemming/scratch/l/lause/RNA/snpseq01320/work/90/a5497cea61f34b022dd75e1dc80778 I was trying to get the .exitcode , is it also in the same directory?

This is the problem:

Detected 1 oom_kill event ... Some of the step tasks have been OOM Killed.

where "oom" stands for "out of memory". You need to ask for more memory. From the

.command.run

it looks like you asked for ~56 GiB, so go for 110 or so. In

.command.sh

you should be able to see how much was asked for by Java, in a block looking like this:

bbnorm.sh \\
        $input \\
        $output \\
        $args \\
        threads=64 \\
        -Xmx48g \\
        &> ${prefix}.bbnorm.log

(Everything starting with a

$

also being replaced.) The key is the

-Xmx48g

string. The actual number there should be ~80% of the memory you asked for. And, yes, the

.exitcode

file should be in the same directory. Not needed in this case.

I do not have any number there at all: out=all_samples.fastq.gz \ target=100 min=5 \ threads=64 \ -Xmx44g \ and its not 48 but 44. Could that be a problem as well?

44 makes sense. It should be 80% of the total you asked for which was 55. Try increasing this to 110 in a config file, but keep the cpu setting.

So I increase the memory but keep the cpu. Just not sure where the 55 comes from? The memory was 120, wasnt it?

No, you must have asked for 55 as it says that in

.command.run

. Maybe an error in the config file?

withName: BBMAP_BBNORM {
  memory = 110.GB
  cpus = 12
  time = '3d'
}

Or something...

I had memory on 120 and cpu on 48 for 3 days, now I have memory on 250 and cpu on 48 with time 2 days. it seems to run now. I am on a shared partition, been running for 6 min.

No, there must be some mistake somewhere if the

.command.run

you pasted is correct. That says 55 GiB and 64 cpus. You need to have

withName: BBMAP_BBNORM

.

that might be my problem 😄 I do not have withName: BBMAP_BBNORM but BBNORM

I think I would. You asked for 110 GiB? I recently ran a few very large samples on 120 GiB (nomially), but that was on a sample-by-sample basis. So, try doubling. Does it fail right away or after a while. (You can compare the timestamps of

.command.sh

and `.command.log`:

ls -l .command.sh .command.log

The former is created when the job starts, the second is written to during execution.)

There is 1 day and 2 minutes inbetween: Aug 23 12:49 .command.log Aug 22 12:47 .command.sh and in the command.log I get the OOM: " error: Detected 1 oom_kill event in StepId=11868437.batch. Some of the step tasks have been OOM Killed." So I guess out of memory is the problem? Can it happen that is starts and at some point runs out of it? E.g. because the amount of samples?

Absolutely. You have too much data for assembly, but also too much for normalization -- difficult. Recently, I've gotten a couple of 1 TiB assembly jobs through the queue -- maybe worth trying without bbnorm again since you got more compute time? (A three day 1 TiB node process is still 18432 compute hours so way beyond what you have. The project I'm running this on has 30,000 hours, but several jobs start in parallel.)

So maybe I just start the 15 water samples and not the 33 sediment ones. And hope I can get those to run.

That's a possibility. If you don't co-assemble, they will be more difficult to compare. There are manual ways of doing this though through clustering. (Which we might put into the pipeline given enough user interest plus some testing.)

what would you suggest to run megahit assembly on? right now I have 48 cpus and 440GB for 4days. Yes I have to run them separately anyway as they have been sequenced on 2 different projects. For now I think it will be fine just to get the data done. Potential samples of interest might be ran for co-assemble at a later stage if necessary (for now I would be happy to have first some data at all :D)

For "normal" data 440 GiB should be fine -- too much rather than too little, but sequence depths have become crazy -- so if you haven't yet tried that with the water samples only, then do that.

I tried so many things, so I might have already tried it. I have a range between 52 million and 149 million read pairs per sample in seq. depth. only for the water samples, the sediment ones are less but are double amount of samples 😄

I know... Try 440. And if it fails, increase

will do thanks 🙂

The easy thing first: Your cpu allocation isn't calculated per calendar month, but running 30 days. Which of course is not much comfort for you.

Ah ok, then I might run into the problem that my contract ends before I get the allocation again 😄 But then there is nothing I can do about it. I asked support already but they are unfortunately fairly slow replying. How much more do you think should I ask for?

Memory for Megahit? You might as well ask for 1 TiB, i.e. 1000 GiB. (Because of the problems with SU VPN and KTH.se I can't check what I have now, but if you ask for too much the pipeline will fail right away telling you asked for more than you can. Then just tune it down a bit and try again.

Hi, thanks for the update, lets hope it will start then 🙂