https://nf-co.re logo
Join Slack
Powered by
Hi, I am running an nf-core/metatdenovo workflow o...
# metatdenovo
b
Hi, I am running an nf-core/metatdenovo workflow on the HPC system, and the BBNorm step fails after approximately 41 hours while trying to write its normalized FASTQ output. The job successfully completed the k-mer counting/hash-table construction: • BBNorm memory allocation: 819 GB • Hash table: ~598 GB • Estimated unique k-mers: ~211 billion • Hash-table occupancy: 86.08% • Table creation time: ~147,067 s (~41 h) The process then fails with: java.io.IOException: Cannot run program “bgzip”: /scratch/SlurmTMP/userID.23778641/TEMPFILE_BBNORM_P1_R1_11cf4e18dc999d13e904e68b23d0cc__62875.239315649414.fq.gz (No such file or directory) Caused by: java.io.FileNotFoundException: /scratch/SlurmTMP/userID.23778641/TEMPFILE_BBNORM_P1_R1_11cf4e18dc999d13e904e68b23d0cc__62875.239315649414.fq.gz (No such file or directory) Exception in thread “main” java.lang.AssertionError: Could not execute bgzip -c -f -l 4 -@ 6 The BBNorm command is: bbnorm.sh in=<~100 FASTQ files> out=all_samples.fastq.gz target=50 min=5 threads=6 -Xmx819g The process is running inside the nf-core/metatdenovo Singularity container. I have verified that bgzip is present inside the container: /opt/conda/bin/bgzip bgzip (htslib) 1.21 and /opt/conda/bin is in the container PATH. BBNorm itself does not explicitly set a temporary directory. The error indicates that it is attempting to use: /scratch/SlurmTMP/userID.23778641/ for a temporary FASTQ file. An initial check shows no node or filesystem problems on the node used. Could this be a Singularity problem (i.e. that you have to mount/connect the external directories to some internal container overlays)?
d
Right. I'll look into this. 1. To rescue the files you have so you don't have to rerun BBNorm (might mean a separate Megahit process) and 2. to make sure the pipeline is fixed.
Can you post 
.nextflow.log
from the directory in which you run the pipeline, plus the 
.command.*
files from the process' work directory?
b
.commnad.sh
Copy code
#!/usr/bin/env bash -C -e -u -o pipefail
bbnorm.sh \
    in=26Feb1256-C03.processed.fastq.gz,26Feb1256-A05.processed.fastq.gz,26Feb1256-C02.processed.fastq.gz,26Feb1256-B05.processed.fastq.gz,26Feb1256-E07.processed.fastq.gz,26Feb1256-D02.processed.fastq.gz,26Feb1256-C08.processed.fastq.gz,26Feb1256-A09.processed.fastq.gz,26Feb1256-E01.processed.fastq.gz,26Feb1256-D01.processed.fastq.gz,26Feb1256-E04.processed.fastq.gz,26Feb1256-H03.processed.fastq.gz,26Feb1256-E08.processed.fastq.gz,26Feb1256-B08.processed.fastq.gz,26Feb1256-A07.processed.fastq.gz,26Feb1256-C05.processed.fastq.gz,26Feb1256-H02.processed.fastq.gz,26Feb1256-E02.processed.fastq.gz,26Feb1256-G02.processed.fastq.gz,26Feb1256-E05.processed.fastq.gz,26Feb1256-H06.processed.fastq.gz,26Feb1256-G04.processed.fastq.gz,26Feb1256-F05.processed.fastq.gz,26Feb1256-B09.processed.fastq.gz,26Feb1256-E03.processed.fastq.gz,26Mar681-H01.processed.fastq.gz,26Feb1256-B03.processed.fastq.gz,26Feb1256-A10.processed.fastq.gz,26Feb1256-D03.processed.fastq.gz,26Feb1256-G08.processed.fastq.gz,26Feb1256-F08.processed.fastq.gz,26Feb1256-D06.processed.fastq.gz,26Feb1256-A04.processed.fastq.gz,26Feb1256-C06.processed.fastq.gz,26Feb1256-H04.processed.fastq.gz,26Feb1256-D09.processed.fastq.gz,26Feb1256-E06.processed.fastq.gz,26Feb1256-H01.processed.fastq.gz,26Feb1256-A01.processed.fastq.gz,26Feb1256-F07.processed.fastq.gz,26Feb1256-A02.processed.fastq.gz,26Feb1256-B01.processed.fastq.gz,26Feb1256-A06.processed.fastq.gz,26Feb1256-G07.processed.fastq.gz,26Feb1256-H05.processed.fastq.gz,26Feb1256-C07.processed.fastq.gz,26Feb1256-B04.processed.fastq.gz,26Mar681-D01.processed.fastq.gz,26Feb1256-F10.processed.fastq.gz,26Feb1256-F03.processed.fastq.gz,26Feb1256-H07.processed.fastq.gz,26Feb1256-G06.processed.fastq.gz,26Feb1256-D05.processed.fastq.gz,26Feb1256-H08.processed.fastq.gz,26Feb1256-F04.processed.fastq.gz,26Feb1256-B02.processed.fastq.gz,26Mar681-C03.processed.fastq.gz,26Mar681-E04.processed.fastq.gz,26Mar681-E03.processed.fastq.gz,26Mar681-C02.processed.fastq.gz,26Mar681-F02.processed.fastq.gz,26Mar681-B02.processed.fastq.gz,26Mar681-D03.processed.fastq.gz,26Mar681-A02.processed.fastq.gz,26Mar681-H03.processed.fastq.gz,26Mar681-A04.processed.fastq.gz,26Feb1256-D04.processed.fastq.gz,26Mar681-G04.processed.fastq.gz,26Feb1256-B07.processed.fastq.gz,26Feb1256-A08.processed.fastq.gz,26Mar681-D04.processed.fastq.gz,26Feb1256-G09.processed.fastq.gz,26Mar681-F04.processed.fastq.gz,26Mar681-B04.processed.fastq.gz,26Feb1256-F02.processed.fastq.gz,26Feb1256-E09.processed.fastq.gz,26Feb1256-F06.processed.fastq.gz,26Feb1256-E11.processed.fastq.gz,26Feb1256-F09.processed.fastq.gz,26Feb1256-G03.processed.fastq.gz,26Feb1256-D07.processed.fastq.gz,26Feb1256-C10.processed.fastq.gz,26Feb1256-C01.processed.fastq.gz,26Feb1256-C09.processed.fastq.gz,26Feb1256-D08.processed.fastq.gz,26Mar681-G01.processed.fastq.gz,26Feb1256-A03.processed.fastq.gz,26Mar681-B03.processed.fastq.gz,26Feb1256-C11.processed.fastq.gz,26Mar681-G03.processed.fastq.gz,26Feb1256-D10.processed.fastq.gz,26Feb1256-H09.processed.fastq.gz,26Feb1256-C04.processed.fastq.gz,26Feb1256-A11.processed.fastq.gz,26Feb1256-B06.processed.fastq.gz,26Feb1256-F01.processed.fastq.gz,26Feb1256-G01.processed.fastq.gz,26Feb1256-G05.processed.fastq.gz,26Mar681-F03.processed.fastq.gz,26Feb1256-G10.processed.fastq.gz,26Feb1256-D11.processed.fastq.gz,26Feb1256-B10.processed.fastq.gz,26Mar681-G02.processed.fastq.gz,26Mar681-E05.processed.fastq.gz,26Mar681-A05.processed.fastq.gz,26Mar681-A01.processed.fastq.gz,26Mar681-B05.processed.fastq.gz,26Mar681-A03.processed.fastq.gz,26Mar681-C01.processed.fastq.gz,26Feb1256-H10.processed.fastq.gz,26Mar681-E02.processed.fastq.gz,26Mar681-G05.processed.fastq.gz,26Feb1256-E10.processed.fastq.gz,26Mar681-F05.processed.fastq.gz,26Mar681-H05.processed.fastq.gz,26Mar681-H04.processed.fastq.gz,26Mar681-A06.processed.fastq.gz,26Mar681-E01.processed.fastq.gz,26Feb1256-B11.processed.fastq.gz,26Mar681-D05.processed.fastq.gz,26Mar681-H02.processed.fastq.gz,26Mar681-D02.processed.fastq.gz,26Mar681-F01.processed.fastq.gz,26Mar681-C04.processed.fastq.gz,26Mar681-C05.processed.fastq.gz,26Mar681-B01.processed.fastq.gz \
    out=all_samples.fastq.gz \
    target=50 min=5 \
    threads=6 \
    -Xmx819g \
    &> all_samples.bbnorm.log

# capture process environment
set +u
set +e
cd "$NXF_TASK_WORKDIR"

nxf_eval_cmd() {
    {
        IFS=$'\n' read -r -d '' "${1}";
        IFS=$'\n' read -r -d '' "${2}";
        (IFS=$'\n' read -r -d '' _ERRNO_; return ${_ERRNO_});
    } < <((printf '\0%s\0%d\0' "$(((({ shift 2; "${@}"; echo "${?}" 1>&3-; } | tr -d '\0' 1>&4-) 4>&2- 2>&1- | tr -d '\0' 1>&4-) 3>&1- | exit "$(cat)") 4>&1-)" "${?}" 1>&2) 2>&1)
}

echo '' > .command.env
#
nxf_eval_cmd STDOUT STDERR bash -c "bbversion.sh | grep -v \"Duplicate cpuset\""
status=$?
if [ $status -eq 0 ]; then
  echo nxf_out_eval_22="$STDOUT" >> .command.env
  echo /nxf_out_eval_22/=exit:0 >> .command.env
else
  echo nxf_out_eval_22="$STDERR" >> .command.env
  echo /nxf_out_eval_22/=exit:$status >> .command.env
fi
.nextflow.log
Copy code
Sep-12 06:37:31.435 [TaskFinalizer-1] DEBUG nextflow.processor.TaskProcessor - Handling unexpected condition for
  task: name=NFCORE_METATDENOVO:METATDENOVO:BBMAP_BBNORM (all_samples); work-dir=/gxfs_work/work/c0/93ee8435fa7bf1fcbcaa9f752f2232
  error [nextflow.exception.ProcessFailedException]: Process `NFCORE_METATDENOVO:METATDENOVO:BBMAP_BBNORM (all_samples)` terminated with an error exit status (1)
Sep-12 06:37:31.501 [TaskFinalizer-1] ERROR nextflow.processor.TaskProcessor - Error executing process > 'NFCORE_METATDENOVO:METATDENOVO:BBMAP_BBNORM (all_samples)'

Caused by:
  Process `NFCORE_METATDENOVO:METATDENOVO:BBMAP_BBNORM (all_samples)` terminated with an error exit status (1)

Command executed:

  bbnorm.sh \
      in=26Feb1256-C03.processed.fastq.gz,26Feb1256-A05.processed.fastq.gz,26Feb1256-C02.processed.fastq.gz,26Feb1256-B05.processed.fastq.gz,26Feb1256-E07.processed.fastq.gz,26Feb1256-D02.processed.fastq.
gz,26Feb1256-C08.processed.fastq.gz,26Feb1256-A09.processed.fastq.gz,26Feb1256-E01.processed.fastq.gz,26Feb1256-D01.processed.fastq.gz,26Feb1256-E04.processed.fastq.gz,26Feb1256-H03.processed.fastq.gz,26F
eb1256-E08.processed.fastq.gz,26Feb1256-B08.processed.fastq.gz,26Feb1256-A07.processed.fastq.gz,26Feb1256-C05.processed.fastq.gz,26Feb1256-H02.processed.fastq.gz,26Feb1256-E02.processed.fastq.gz,26Feb1256
-G02.processed.fastq.gz,26Feb1256-E05.processed.fastq.gz,26Feb1256-H06.processed.fastq.gz,26Feb1256-G04.processed.fastq.gz,26Feb1256-F05.processed.fastq.gz,26Feb1256-B09.processed.fastq.gz,26Feb1256-E03.p
rocessed.fastq.gz,26Mar681-H01.processed.fastq.gz,26Feb1256-B03.processed.fastq.gz,26Feb1256-A10.processed.fastq.gz,26Feb1256-D03.processed.fastq.gz,26Feb1256-G08.processed.fastq.gz,26Feb1256-F08.processe
d.fastq.gz,26Feb1256-D06.processed.fastq.gz,26Feb1256-A04.processed.fastq.gz,26Feb1256-C06.processed.fastq.gz,26Feb1256-H04.processed.fastq.gz,26Feb1256-D09.processed.fastq.gz,26Feb1256-E06.processed.fast
q.gz,26Feb1256-H01.processed.fastq.gz,26Feb1256-A01.processed.fastq.gz,26Feb1256-F07.processed.fastq.gz,26Feb1256-A02.processed.fastq.gz,26Feb1256-B01.processed.fastq.gz,26Feb1256-A06.processed.fastq.gz,2
6Feb1256-G07.processed.fastq.gz,26Feb1256-H05.processed.fastq.gz,26Feb1256-C07.processed.fastq.gz,26Feb1256-B04.processed.fastq.gz,26Mar681-D01.processed.fastq.gz,26Feb1256-F10.processed.fastq.gz,26Feb125
6-F03.processed.fastq.gz,26Feb1256-H07.processed.fastq.gz,26Feb1256-G06.processed.fastq.gz,26Feb1256-D05.processed.fastq.gz,26Feb1256-H08.processed.fastq.gz,26Feb1256-F04.processed.fastq.gz,26Feb1256-B02.
processed.fastq.gz,26Mar681-C03.processed.fastq.gz,26Mar681-E04.processed.fastq.gz,26Mar681-E03.processed.fastq.gz,26Mar681-C02.processed.fastq.gz,26Mar681-F02.processed.fastq.gz,26Mar681-B02.processed.fa
stq.gz,26Mar681-D03.processed.fastq.gz,26Mar681-A02.processed.fastq.gz,26Mar681-H03.processed.fastq.gz,26Mar681-A04.processed.fastq.gz,26Feb1256-D04.processed.fastq.gz,26Mar681-G04.processed.fastq.gz,26Feb1256-B07.processed.fastq.gz,26Feb1256-A08.processed.fastq.gz,26Mar681-D04.processed.fastq.gz,26Feb1256-G09.processed.fastq.gz,26Mar681-F04.processed.fastq.gz,26Mar681-B04.processed.fastq.gz,26Feb1256-F02.processed.fastq.gz,26Feb1256-E09.processed.fastq.gz,26Feb1256-F06.processed.fastq.gz,26Feb1256-E11.processed.fastq.gz,26Feb1256-F09.processed.fastq.gz,26Feb1256-G03.processed.fastq.gz,26Feb1256-D07.processed.fastq.gz,26Feb1256-C10.processed.fastq.gz,26Feb1256-C01.processed.fastq.gz,26Feb1256-C09.processed.fastq.gz,26Feb1256-D08.processed.fastq.gz,26Mar681-G01.processed.fastq.gz,26Feb1256-A03.processed.fastq.gz,26Mar681-B03.processed.fastq.gz,26Feb1256-C11.processed.fastq.gz,26Mar681-G03.processed.fastq.gz,26Feb1256-D10.processed.fastq.gz,26Feb1256-H09.processed.fastq.gz,26Feb1256-C04.processed.fastq.gz,26Feb1256-A11.processed.fastq.gz,26Feb1256-B06.processed.fastq.gz,26Feb1256-F01.processed.fastq.gz,26Feb1256-G01.processed.fastq.gz,26Feb1256-G05.processed.fastq.gz,26Mar681-F03.processed.fastq.gz,26Feb1256-G10.processed.fastq.gz,26Feb1256-D11.processed.fastq.gz,26Feb1256-B10.processed.fastq.gz,26Mar681-G02.processed.fastq.gz,26Mar681-E05.processed.fastq.gz,26Mar681-A05.processed.fastq.gz,26Mar681-A01.processed.fastq.gz,26Mar681-B05.processed.fastq.gz,26Mar681-A03.processed.fastq.gz,26Mar681-C01.processed.fastq.gz,26Feb1256-H10.processed.fastq.gz,26Mar681-E02.processed.fastq.gz,26Mar681-G05.processed.fastq.gz,26Feb1256-E10.processed.fastq.gz,26Mar681-F05.processed.fastq.gz,26Mar681-H05.processed.fastq.gz,26Mar681-H04.processed.fastq.gz,26Mar681-A06.processed.fastq.gz,26Mar681-E01.processed.fastq.gz,26Feb1256-B11.processed.fastq.gz,26Mar681-D05.processed.fastq.gz,26Mar681-H02.processed.fastq.gz,26Mar681-D02.processed.fastq.gz,26Mar681-F01.processed.fastq.gz,26Mar681-C04.processed.fastq.gz,26Mar681-C05.processed.fastq.gz,26Mar681-B01.processed.fastq.gz \
      out=all_samples.fastq.gz \
      target=50 min=5 \
      threads=6 \
      -Xmx819g \
      &> all_samples.bbnorm.log

Command exit status:
  1

Command output:
  (empty)

Work dir:
  /gxfs_work/work/c0/93ee8435fa7bf1fcbcaa9f752f2232

Container:
  /gxfs_home/.singularity/community-cr-prod.seqera.io-docker-registry-v2-blobs-sha256-5a-5aae5977ff9de3e01ff962dc495bfa23f4304c676446b5fdf2de5c7edfa2dc4e-data.img

Tip: view the complete command output by changing to the process work dir and entering the command `cat .command.out`
Sep-12 06:37:31.524 [TaskFinalizer-1] INFO  nextflow.Session - Execution cancelled -- Finishing pending tasks before exit
Sep-12 06:37:31.528 [Actor Thread 162] DEBUG nextflow.container.SingularityCache - Singularity found local store for image=<https://depot.galaxyproject.org/singularity/mulled-v2-0f92c152b180c7cd39d9b0e6822f8c89ccb59c99:8ec213d21e5d03f9db54898a2baeaf8ec729b447-0>; path=/gxfs_home/.singularity/depot.galaxyproject.org-singularity-mulled-v2-0f92c152b180c7cd39d9b0e6822f8c89ccb59c99-8ec213d21e5d03f9db54898a2baeaf8ec729b447-0.img
Sep-12 06:37:31.566 [TaskFinalizer-1] ERROR nextflow.Nextflow - Pipeline failed. Please refer to troubleshooting docs for common issues: <https://nf-co.re/docs/running/troubleshooting>
Sep-12 06:37:31.566 [main] DEBUG nextflow.Session - Session await > all processes finished
Sep-12 06:37:31.566 [Task monitor] DEBUG n.processor.TaskPollingMonitor - <<< barrier arrives (monitor: slurm) - terminating tasks monitor poll loop
Sep-12 06:37:31.567 [main] DEBUG nextflow.Session - Session await > all barriers passed
Sep-12 06:37:31.631 [Actor Thread 79] DEBUG nextflow.sort.BigSort - Sort completed -- entries: 1; slices: 1; internal sort time: 0.0 s; external sort time: 0.001 s; total time: 0.001 s
Sep-12 06:37:31.633 [Actor Thread 79] DEBUG nextflow.file.FileCollector - >> temp file exists? true
Sep-12 06:37:31.636 [Actor Thread 79] DEBUG nextflow.file.FileCollector - Retrieved cached collect-files from: /gxfs_work/work/collect-file/b313f8b9eacfb264475ad450979ebc39
Sep-12 06:37:31.636 [Actor Thread 79] DEBUG nextflow.file.FileCollector - Deleting file collector temp dir: /tmp/nxf-8040013573125260202
Sep-12 06:37:31.638 [main] DEBUG nextflow.util.ThreadPoolManager - Thread pool 'TaskFinalizer' shutdown completed (hard=false)
Sep-12 06:37:31.639 [main] DEBUG nextflow.util.ThreadPoolManager - Thread pool 'PublishDir' shutdown completed (hard=false)
Sep-12 06:37:31.641 [main] INFO  nextflow.Nextflow - -ESC[0;35m[nf-core/metatdenovo]ESC[0;31m Pipeline completed with errorsESC[0m-
Sep-12 06:37:31.645 [main] DEBUG n.trace.WorkflowStatsObserver - Workflow completed > WorkflowStats[succeededCount=0; failedCount=1; ignoredCount=0; cachedCount=378; pendingCount=1; submittedCount=0; runningCount=0; retriesCount=0; abortedCount=0; succeedDuration=0ms; failedDuration=10d 5h 7m 54s; cachedDuration=22d 15h 40m 54s;loadCpus=0; loadMemory=0; peakRunning=1; peakCpus=6; peakMemory=1 TB; ]
Sep-12 06:37:31.823 [main] DEBUG nextflow.trace.ReportObserver - Workflow completed -- rendering execution report
Sep-12 06:37:32.313 [main] DEBUG nextflow.trace.TimelineObserver - Workflow completed -- rendering execution timeline
Sep-12 06:37:32.426 [main] DEBUG nextflow.trace.TraceFileObserver - Workflow completed -- saving trace file
Sep-12 06:37:32.670 [main] DEBUG nextflow.cache.CacheDB - Closing CacheDB done
Sep-12 06:37:32.686 [main] INFO  org.pf4j.AbstractPluginManager - Stop plugin 'nf-schema@2.7.2'
Sep-12 06:37:32.686 [main] DEBUG nextflow.plugin.BasePlugin - Plugin stopped nf-schema
Sep-12 06:37:32.686 [main] DEBUG nextflow.util.ThreadPoolManager - Thread pool 'FileTransfer' shutdown completed (hard=false)
Sep-12 06:37:32.686 [main] DEBUG nextflow.script.ScriptRunner - > Execution complete -- Goodbye
d
You can attach the files themselves by clicking the +-sign. We particularly need 
.command.run
and 
.command.env
but also 
.command.log
and 
.command.err
.
(It seems you were right: The temporary directory was not mounted. Unfortunately, it also seems what BBNorm had done never reached disk, so can't be rescued.)
b
.command.run
.command.run
.command.sh
.command.sh
.nextflow.log
.log and .err are empty
d
How about 
.command.env
-- is there one?
b
No, there is no .env
.begin, .err, .log, .trace are empty and only .run and .sh are populated
👍 1
d
Which version of the pipeline did you run? 
-r dev
?
It was the released 1.4.0. Working on a patch release for this. You'll get a 1.4.1 version to resume from.
b
v 1.4.0
d
2 Views
Open in Slack
PreviousNext
Powered by