Hi everyone, some days ago I am having problems la...
# taxprofiler
j
Hi everyone, some days ago I am having problems launching nf-core/taxprofiler on CESGA (HPC which uses SLURM as execurtor). I am using only one database and 4 samples to test the pipeline. When I lauch the script (sbatch) the process beguing well, FASQC ends with no problems, but when the first sample is analized with Kraken2 the pipeline fails, it becomes to retry and retry, but it never end. Here is an example: executor > slurm (18) [- ] NFC…PROFILERtaxprofilerUNTAR - [b6/08c134] NFC…LERtaxprofilerFASTQC (3) | 4 of 4, cached: 4 ✔️ [2b/b35890] NFC…AKEN2 (standard|kraken2|2) | 0 of 8, retries: 16 [- ] NFC…PROFILINGBRACKEN_BRACKEN - [- ] NFC…OFILERtaxprofilerMULTIQC - [db/3a39af] NOTE: Process
NFCORE_TAXPROFILER:taxprofiler:PROFILING:KRAKEN2_KRAKEN2 (standard|bracken|4)
terminated for an unknown reason -- Likely it has been terminated by the external system -- Execution is retried (2)
When the job ends, the script.out shows that: ERROR ~ Error executing process > 'NFCORE_TAXPROFILERtaxprofilerPROFILING:KRAKEN2_KRAKEN2 (standard|kraken2|1)' Caused by: Process
NFCORE_TAXPROFILER:taxprofiler:PROFILING:KRAKEN2_KRAKEN2 (standard|kraken2|1)
terminated for an unknown reason -- Likely it has been terminated by the external system Command executed: kraken2 \ --db k2_standard_20250402 \ --threads 16 \ --report 1_1_standard.kraken2.kraken2.report.txt \ --gzip-compressed \ \ \ --output /dev/null \ --paired \ --quick \ 1_1.fq.gz 1_2.fq.gz cat <<-END_VERSIONS > versions.yml "NFCORE_TAXPROFILERtaxprofilerPROFILING:KRAKEN2_KRAKEN2": kraken2: $(echo $(kraken2 --version 2>&1) | sed 's/^.*Kraken version //; s/ .*$//') pigz: $( pigz --version 2>&1 | sed 's/pigz //g' ) END_VERSIONS Command exit status: - Command output: (empty) Command wrapper: slurmstepd: info: Setting TMPDIR to /scratch/920716. Previous errors about TMPDIR can be discarded Loading database information... Work dir: /mnt/lustre/scratch/nlsas/home/csic/bse/mmp/proyect_severo_ochoa/nextflow_work/d6/cdeba465fe28c10c3e98128045c996 Container: /mnt/lustre/scratch/nlsas/home/csic/bse/mmp/proyect_severo_ochoa/apptainer_cache/community.wave.seqera.io-library-kraken2_coreutils_pigz-45764814c4bb5bf3.img Tip: when you have fixed the problem you can continue the execution adding the option
-resume
to the run command line -- Check '.nextflow.log' file for details ERROR ~ Pipeline failed. Please refer to troubleshooting docs: https://nf-co.re/docs/usage/troubleshooting -- Check '.nextflow.log' file for details [86/96cd10] Re-submitted process > NFCORE_TAXPROFILERtaxprofilerPROFILING:KRAKEN2_KRAKEN2 (standard|kraken2|3)
I've checked the directory especified at the end of the script.out and there I can see the kraken report and other files: cd ./nextflow_work/86/96cd10faf323c5bed454fe652e7f7a ll -a total 1232 drwxr-xr-x 2 * 4096 sep 16 21:11 . drwxr-xr-x 4 * 4096 sep 16 19:01 .. -rw-r--r-- 1 * 1211243 sep 16 21:11 3_3_standard.kraken2.kraken2.report.txt -rw-r--r-- 1 * 0 sep 16 20:02 .command.begin -rw-r--r-- 1 * 624 sep 16 21:11 .command.err -rw-r--r-- 1 * 1482 sep 16 21:11 .command.log -rw-r--r-- 1 * 0 sep 16 21:11 .command.out -rw-r--r-- 1 * 12721 sep 16 19:41 .command.run -rw-r--r-- 1 * 526 sep 16 19:41 .command.sh -rw-r--r-- 1 * 307 sep 16 21:11 .command.trace -rw-r--r-- 1 * 1 sep 16 21:11 .exitcode -rw-r--r-- 1 * 93 sep 16 21:11 versions.yml If we inspect the .command.err we can see: cat .command.err Loading database information... done. 32032736 sequences (9609.82 Mbp) processed in 181.492s (10589.8 Kseq/m, 3176.94 Mbp/m). 28169576 sequences classified (87.94%) 3863160 sequences unclassified (12.06%) ls: no se puede acceder a '*.classified.*': No existe el fichero o el directorio ls: no se puede acceder a '*.classified_*': No existe el fichero o el directorio ls: no se puede acceder a '*.unclassified.*': No existe el fichero o el directorio ls: no se puede acceder a '*.unclassified_*': No existe el fichero o el directorio ls: no se puede acceder a '*classifiedreads.txt': No existe el fichero o el directorio I don't know what's happening, I hope someone could help me, if anyone need some more information, please ask me, thank you all.
j
Hi @Juan Serranoi Fernández!
In the future please could you wrap error message/scode in triple backticks
markdownhelp
s
Using some code formatting helps with readability and finding the issue. Here are some tips about how to format code snippets nicely in Slack: http://tiny.cc/mdcodeblocks http://tiny.cc/slacksnippets
j
Makes itrto read 🙂
But otherwise I think you have to errors, first
Copy code
Caused by:
  Process `NFCORE_TAXPROFILER:taxprofiler:PROFILING:KRAKEN2_KRAKEN2 (standard|kraken2|1)` terminated for an unknown reason -- Likely it has been terminated by the external system
I guess is caused by SLURM killing the job as it took too long But this is maybe related due to
Copy code
ls: no se puede acceder a '*.classified.*': No existe el fichero o el directorio
ls: no se puede acceder a '*.classified_*': No existe el fichero o el directorio
ls: no se puede acceder a '*.unclassified.*': No existe el fichero o el directorio
ls: no se puede acceder a '*.unclassified_*': No existe el fichero o el directorio
ls: no se puede acceder a '*classifiedreads.txt': No existe el fichero o el directorio
As you identified. Which I think is coming from the the tool somehow not being able to write the output files...? This would be something to do with your system I think. But I see that your
.command.log
(which combines
.std
and
out
in one) is larger... could you send the
.command.loig
file and see what other information is in there?
j
Hi James, thank you for your help. I'll take notes of your advices. Here is the .command.log
Copy code
cat .command.log
slurmstepd: info: Setting TMPDIR to /scratch/921548. Previous errors about TMPDIR can be discarded
Loading database information... done.
32032736 sequences (9609.82 Mbp) processed in 181.492s (10589.8 Kseq/m, 3176.94 Mbp/m).
  28169576 sequences classified (87.94%)
  3863160 sequences unclassified (12.06%)
ls: no se puede acceder a '*.classified.*': No existe el fichero o el directorio
ls: no se puede acceder a '*.classified_*': No existe el fichero o el directorio
ls: no se puede acceder a '*.unclassified.*': No existe el fichero o el directorio
ls: no se puede acceder a '*.unclassified_*': No existe el fichero o el directorio
ls: no se puede acceder a '*classifiedreads.txt': No existe el fichero o el directorio

*****************************************************************************
*                                                                           *
*                    JOB EFFICIENCY REPORT (seff 921548)                   *
*                                                                           *
*****************************************************************************

Job ID: 921548
Cluster: finisterrae3
User/Group: csbsemmp/csic
State: COMPLETED (exit code 0)
Nodes: 1
Cores per node: 16
CPU Utilized: 00:10:21
CPU Efficiency: 0.94% of 18:23:28 core-walltime
Job Wall-clock time: 01:08:58
Memory Utilized: 90.92 GB
Memory Efficiency: 75.77% of 120.00 GB

*****************************************************************************
j
Hrmm, the only thing I can think of s your scheduler killed something or there was an IO hiccup for the tool not to find it's own files 😞
I guess the first thing is to try
-resume
but also with a config that increases your initial amount of memory
resourcehelp
s
You likely need to configure the pipeline to your available computational resources to you machine or infrastructure. Please see the following two(!) sections: - https://nf-co.re/docs/usage/getting_started/configuration#tuning-workflow-resources - https://nf-co.re/docs/usage/getting_started/configuration#max-resources of the documentation for more information.
j
(specifically for Kraken)
j
Here is my nextflow.config file. If you can help me to configure it I will be so glad.
Copy code
process {
  executor = 'slurm'
  queue = 'medium'
  cpus = 4
  memory = '8 GB'
  time = '4h'
  stageInMode = 'symlink'
  stageOutMode = 'copy'
  scratch = true
  errorStrategy = 'retry'
  maxRetries = 2
  maxForks = 2

  withName: 'NFCORE_TAXPROFILER:TAXPROFILER:FASTQC' {
    memory = '6144 MB'
    time = '1h'
    cpus = 2
  }

  withName: 'NFCORE_TAXPROFILER:TAXPROFILER:PROFILING:KRAKEN2_KRAKEN2' {
    cpus = 16
    memory = '120 GB'
    time = '6h'
    clusterOptions = ''
  }

  withName: 'NFCORE_TAXPROFILER:TAXPROFILER:PROFILING:BRACKEN_BRACKEN' {
    cpus = 8
    memory = '24 GB'
    time = '3h'
    clusterOptions = ''
  }
}

params {
  input {
    staging = false
  }
}

executor {
  queueSize = 50
  queueStatusPollingInterval = '10 sec'
  exitStatusReadTimeout = '5 min'
}

apptainer {
  enabled = true
  autoMounts = false
  runOptions = '--bind /mnt/lustre'
  cacheDir = 'path/apptainer_cache'
}

env {
  APPTAINER_BINDPATH = '/path/'   // Asegura que apptainer monte esta ruta
}
Also, I will post my nextflow pipeline:
Copy code
nextflow run nf-core/taxprofiler \
    -r 1.2.4 \
    -profile apptainer \
    --input /path/reduced_samplesheet.csv \
    --databases /path/reduced_databases.csv \
    --outdir /path/reduced_results/ \
    --run_kraken2 \
    --run_bracken true \
    -resume
j
Hrmm
Copy code
withName: 'NFCORE_TAXPROFILER:TAXPROFILER:PROFILING:KRAKEN2_KRAKEN2' {
    cpus = 16
    memory = '120 GB'
    time = '6h'
    clusterOptions = ''
  }
Looks good, is it still retrying with this? But maybe you could explicitly reference the confic file in the command too by adding
-c nextflow.config
before resume
j
I am submitting the job and executing the pipeline from the same directory where I have the file nextflow.config; by the way, I will add the flag. If I find something new I will let you know it, thank you very much James.
👍 1
Hi everyone, I think that I found the problem which did not let me lauch propertly the pipeline on SLURM. When I changed my database (88 GB) to a smaller one (8 GB) the process ended perfectly, so if anyone has a problem similar to me I recommend to use so much RAM, CPUs and time, as much as you can, because my porbe for 4 samples and a database of 8GB lasted 3h 31min and used 12,4 CPUs, so do not be stingy with the resources. Thank you all, especially @James Fellows Yates
👍 1
j
Yeah, metagenomics is always going to be RAM heavy 😞