https://nf-co.re logo
Join Slack
Powered by
Hi, I keep getting and error 140 in the Multiqc st...
# help
c
Hi, I keep getting and error 140 in the Multiqc step of the taxprofiler pipeline. Anybody seen this before? Any hint?
j
What is the full error message?
c
Sep-06 115343.382 [TaskFinalizer-2] DEBUG nextflow.processor.TaskProcessor - Handling unexpected condition for task: name=NFCORE_TAXPROFILERtaxprofilerMULTIQC; work-dir=/nfs2/donatic/projects/Forest_soil_2/taxprofiler/work/20/6129e458d1485c1992010683ff9ee3 error [nextflow.exception.ProcessFailedException]: Process 
NFCORE_TAXPROFILER:taxprofiler:MULTIQC
terminated with an error exit status (140) Sep-06 115343.415 [TaskFinalizer-2] ERROR nextflow.processor.TaskProcessor - Error executing process > 'NFCORE_TAXPROFILERtaxprofilerMULTIQC' Caused by: Process 
NFCORE_TAXPROFILER:taxprofiler:MULTIQC
terminated with an error exit status (140) Command executed: multiqc \ --force \ --title "multiqc" \ --config multiqc_config.yml \ \ --cl-config 'custom_logo: "nf-core-taxprofiler_logo_custom_light.png"' \ . cat <<-END_VERSIONS > versions.yml "NFCORE_TAXPROFILERtaxprofilerMULTIQC": multiqc: $( multiqc --version | sed -e "s/multiqc, version //g" ) END_VERSIONS Command exit status: 140 Command output: (empty) Command error: /// MultiQC 🔍 | v1.21 | multiqc | MultiQC Version v1.24.1 now available! | multiqc | Report title: multiqc | multiqc | Module(s) in config.run_modules are unknown: minimap2 | multiqc | Only using modules: fastqc, adapterRemoval, fastp, bbduk, prinseqplusplus, porechop, filtlong, bowtie2, samtools, kraken, kaiju, diamond, malt, motu s, custom_content | multiqc | Search path : /nfs2/donatic/projects/Forest_soil_2/taxprofiler/work/20/6129e458d1485c1992010683ff9ee3 | searching | ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100% 132/132 | fastqc | Found 24 reports | fastqc | Found 24 reports | fastp | Found 12 reports | bbduk | Found 12 reports | kraken | Found 24 reports Work dir: /nfs2/donatic/projects/Forest_soil_2/taxprofiler/work/20/6129e458d1485c1992010683ff9ee3 Tip: you can replicate the issue by changing to the process work dir and entering the command 
bash .command.run
Sep-06 115343.418 [main] DEBUG nextflow.Session - Session await > all processes finished Sep-06 115343.419 [TaskFinalizer-2] DEBUG nextflow.Session - Session aborted -- Cause: Process 
NFCORE_TAXPROFILER:taxprofiler:MULTIQC
terminated with an error exit stat us (140) Sep-06 115343.448 [TaskFinalizer-2] ERROR nextflow.Nextflow - Pipeline failed. Please refer to troubleshooting docs: https://nf-co.re/docs/usage/troubleshooting Sep-06 115343.449 [Task monitor] DEBUG n.processor.TaskPollingMonitor - <<< barrier arrives (monitor: sge) - terminating tasks monitor poll loop Sep-06 115343.449 [main] DEBUG nextflow.Session - Session await > all barriers passed Sep-06 115343.454 [main] INFO nextflow.Nextflow - -[nf-core/taxprofiler] Pipeline completed with errors- Sep-06 115343.459 [main] DEBUG n.trace.WorkflowStatsObserver - Workflow completed > WorkflowStats[succeededCount=0; failedCount=2; ignoredCount=0; cachedCount=120; pendin gCount=0; submittedCount=0; runningCount=0; retriesCount=1; abortedCount=0; succeedDuration=0ms; failedDuration=35s; cachedDuration=43d 9h 56m 52s;loadCpus=0; loadMemory=0; peakRunning=1; peakCpus=1; peakMemory=2 GB; ] Sep-06 115343.459 [main] DEBUG nextflow.trace.TraceFileObserver - Workflow completed -- saving trace file Sep-06 115343.461 [main] DEBUG nextflow.trace.ReportObserver - Workflow completed -- rendering execution report Sep-06 115344.424 [main] DEBUG nextflow.trace.TimelineObserver - Workflow completed -- rendering execution timeline Sep-06 115344.590 [main] DEBUG nextflow.cache.CacheDB - Closing CacheDB done Sep-06 115344.871 [main] INFO org.pf4j.AbstractPluginManager - Stop plugin 'nf-validation@1.1.3' Sep-06 115344.872 [main] DEBUG nextflow.plugin.BasePlugin - Plugin stopped nf-validation Sep-06 115344.874 [main] DEBUG nextflow.script.ScriptRunner - > Execution complete -- Goodbye
this is what I hav eat the bottom of the .nextflow.log file
j
Èxit code140: Typically observed when your cluster is signalling that the job should wrap up because it is about to exceed a time or memory limit. You should speak to your system administrator to help you adjust your job resource requests. See additional guidance below this table.
So you need to increase your wall tile
Tine*
Using a custom config
resourcehelp
s
You likely need to configure the pipeline to your available computational resources to you machine or infrastructure. Please see the following two(!) sections: - https://nf-co.re/docs/usage/getting_started/configuration#tuning-workflow-resources - https://nf-co.re/docs/usage/getting_started/configuration#max-resources of the documentation for more information.
c
But why does it die after no more than 1 min, since most tools have run for hours? I do not see why the cluster should kill this job, and not the other
j
Hrmmmm
Looking now while not feeding my son, it could be insufficient memory
Copy code
For example, the common exit status of 140 indicates that a process failed due to exceeding memory, disk, CPU, or time constraints (for SGE or SLURM clusters). Most frequently, memory is the limiting factor; consider raising the memory allocated for the process in the relevant configuration file.
But the docs links also apply for that 🙂
if it's only 1min that implies to me memory
c
I also had to feed my 22 years old daughter...
😆 1
j
My son is 1yo... but probably similar experience 😆
c
Anyway my params file is like this: { "input": "samplesheet.csv", "databases": "databases.csv", "outdir": "results", "multiqc_title": "multiqc", "perform_shortread_qc": true, "shortread_qc_skipadaptertrim": true, "perform_shortread_complexityfilter": true, "run_kraken2": true, "run_bracken": true, "run_metaphlan": true, "run_motus": true, "motus_use_relative_abundance": true } process { withName: 'NFCORE_TAXPROFILERtaxprofilerFASTQC' { memory = 7.GB } withName: MULTIQC { memory = 40.GB } }
j
Ah, no - params file != config file 🙂
config is for infrastfcuture settings tweaking, params file is for pipeline/software tweaking
You need a separate file that you put the 
process
block in there, and give that file to the pipeline with 
-c <yourconfig>.conf
(in addition to your params file
c
ok, this was not clear to me
I was reading the https://nf-co.re/docs/usage/getting_started/configuration#custom-configuration-files custom configuration file section and I got confuused
"For Nextflow DSL2 nf-core pipelines - parameters defined in the parameter block in 
custom.config
files WILL NOT override defaults in 
nextflow.config
! Please use 
-params-file
in 
yaml
or 
json
format in these cases:"
j
Yeah 
parameters
there are referring to nextflow parameters
which are what you give to the piplein with 
--
e.g. 
--input
or 
--run_kraken2
etc.
I'll have a think how to rephrase that to make it clearer
c
Maybe provide a couple of template files with examples would help
nf-core is great, but fairly complex
j
Have you read the second link yet?
The 'tunning working resources' buit?
Sorry, scrap that question
You have as you've already got the examples in your file you've shared above
It's more of a nextflow thing (as in teh terminoloogy comes from Nextflow rather than nf-core)
But we are in the process of trying to improve our documentation, it just takes time as it's no-ones favourite thing to write 😅
c
Done! It worked. Thanks, I have been going back and forth the whole morning
🎉 1
j
Also feel free to ask future questions about taxprofiler on #C031QH57DSS 🙂
c
Now that I learned that you are there, I'll bug you again. Thanks a lot
j
😬
c
Ok, it took less than expected. I have one more error
Sep-06 152715.847 [TaskFinalizer-1] DEBUG nextflow.processor.TaskProcessor - Handling unexpected condition for task: name=NFCORE_TAXPROFILERtaxprofilerSTANDARDISATION_PROFILES:TAXPASTA_MERGE (motus|db1); work-dir=/nfs2/donatic/projects/Forest_soil_2/taxprofiler/work/e3/fe6df1827 5939ad382c5b5d7469823 error [nextflow.exception.ProcessFailedException]: Process 
NFCORE_TAXPROFILER:taxprofiler:STANDARDISATION_PROFILES:TAXPASTA_MERGE (motus|db1)
terminated with an error e xit status (1) Sep-06 152715.889 [TaskFinalizer-1] ERROR nextflow.processor.TaskProcessor - Error executing process > 'NFCORE_TAXPROFILERtaxprofilerSTANDARDISATION_PROFILES:TAXPASTA_M ERGE (motus|db1)' Caused by: Process 
NFCORE_TAXPROFILER:taxprofiler:STANDARDISATION_PROFILES:TAXPASTA_MERGE (motus|db1)
terminated with an error exit status (1) Command executed: taxpasta merge \ --profiler motus \ --output motus_db1.tsv \ --add-lineage --add-rank-lineage \ --taxonomy taxdump \ \ MATESE-CB-1a_S2_L003_pe_FORESTE_db1.out MATESE-CB-4_S5_L003_pe_FORESTE_db1.out MATESE-CB-1b_S4_L003_pe_FORESTE_db1.out MATESE-CB-3_S8_L003_pe_FORESTE_db1.out MATESE-C B-5_S11_L003_pe_FORESTE_db1.out MATESE-CB-2_S6_L003_pe_FORESTE_db1.out MATESE-FALODE-1b_S10_L003_pe_FORESTE_db1.out MATESE-FALODE-3_S3_L003_pe_FORESTE_db1.out MATESE-FALODE -4_S12_L003_pe_FORESTE_db1.out MATESE-FALODE-1a_S7_L003_pe_FORESTE_db1.out MATESE-FALODE-2_S1_L003_pe_FORESTE_db1.out MATESE-FALODE-5_S9_L003_pe_FORESTE_db1.out cat <<-END_VERSIONS > versions.yml "NFCORE_TAXPROFILERtaxprofilerSTANDARDISATION_PROFILES:TAXPASTA_MERGE": taxpasta: $(taxpasta --version) END_VERSIONS Command exit status: 1 Command output: [152702] CRITICAL Error in sample merge.py:419 'MATESE-CB-1a_S2_L003_pe_FORESTE_db1' with profile 'MATESE-CB-1a_S2_L003_pe_FORESTE_db1.out'. CRITICAL schema_context column check merge.py:424 check_number failure_case index 0 Column read_count dtype('int64') None float64 None Command wrapper: [152702] CRITICAL Error in sample merge.py:419 'MATESE-CB-1a_S2_L003_pe_FORESTE_db1' with profile 'MATESE-CB-1a_S2_L003_pe_FORESTE_db1.out'. CRITICAL schema_context column check merge.py:424 check_number failure_case index 0 Column read_count dtype('int64') None float64 None Work dir: /nfs2/donatic/projects/Forest_soil_2/taxprofiler/work/e3/fe6df18275939ad382c5b5d7469823 Tip: you can replicate the issue by changing to the process work dir and entering the command 
bash .command.run
Sep-06 152715.899 [TaskFinalizer-1] INFO nextflow.Session - Execution cancelled -- Finishing pending tasks before exit Sep-06 152715.903 [TaskFinalizer-1] ERROR nextflow.Nextflow - Pipeline failed. Please refer to troubleshooting docs: https://nf-co.re/docs/usage/troubleshooting Sep-06 152715.904 [main] DEBUG nextflow.Session - Session await > all processes finished Sep-06 152845.911 [Task monitor] DEBUG n.processor.TaskPollingMonitor - Task completed > TaskHandler[jobId: 1558361; id: 133; name: NFCORE_TAXPROFILERtaxprofilerMULTIQC ; status: COMPLETED; exit: 0; error: -; workDir: /nfs2/donatic/projects/Forest_soil_2/taxprofiler/work/ff/3a0d4e6d63f1c411951bd651bfa968 started: 1725629235844; exited: 202 4-09-06T132832.038753Z; ] Sep-06 152845.912 [Task monitor] DEBUG n.processor.TaskPollingMonitor - <<< barrier arrives (monitor: sge) - terminating tasks monitor poll loop Sep-06 152845.912 [main] DEBUG nextflow.Session - Session await > all barriers passed Sep-06 152846.057 [main] DEBUG nextflow.util.ThreadPoolManager - Thread pool 'TaskFinalizer' shutdown completed (hard=false) Sep-06 152851.059 [main] INFO nextflow.util.ThreadPoolHelper - Waiting for file transfers to complete (1 files) Sep-06 152852.713 [main] DEBUG nextflow.util.ThreadPoolManager - Thread pool 'PublishDir' shutdown completed (hard=false) Sep-06 152852.722 [main] INFO nextflow.Nextflow - -[nf-core/taxprofiler] Pipeline completed with errors- Sep-06 152852.727 [main] DEBUG n.trace.WorkflowStatsObserver - Workflow completed > WorkflowStats[succeededCount=1; failedCount=1; ignoredCount=0; cachedCount=131; pendin gCount=0; submittedCount=0; runningCount=0; retriesCount=0; abortedCount=0; succeedDuration=1m 40s; failedDuration=5ms; cachedDuration=43d 9h 59m 7s;loadCpus=0; loadMemory= 0; peakRunning=2; peakCpus=2; peakMemory=552 GB; ] Sep-06 152852.727 [main] DEBUG nextflow.trace.TraceFileObserver - Workflow completed -- saving trace file Sep-06 152852.739 [main] DEBUG nextflow.trace.ReportObserver - Workflow completed -- rendering execution report Sep-06 152853.771 [main] DEBUG nextflow.trace.TimelineObserver - Workflow completed -- rendering execution timeline Sep-06 152854.162 [main] DEBUG nextflow.cache.CacheDB - Closing CacheDB done Sep-06 152854.457 [main] INFO org.pf4j.AbstractPluginManager - Stop plugin 'nf-validation@1.1.3' Sep-06 152854.458 [main] DEBUG nextflow.plugin.BasePlugin - Plugin stopped nf-validation Sep-06 152854.459 [main] DEBUG nextflow.util.ThreadPoolManager - Thread pool 'FileTransfer' shutdown completed (hard=false) Sep-06 152854.460 [main] DEBUG nextflow.script.ScriptRunner - > Execution complete -- Goodbye
I have seen this in a discussion, but could not find a solution
j
Ah, what versio nof the pipeline are you running?
1.1.8?
c
yes
is there a newer one?
j
Poop, OK. I had hoped we had the latest version of taxpasta in that version but I guess we don't. A new major version of the pipeline is coming out next week ™️ that has the lastest version of taxprofiler that shouldn't have that issue
Could you wait that long?
c
Yes, of course. This project will go on for some time, I am starting to look at the data. I'll wait for the new version. Thanks a lot and have a good week end
j
Thank you and likewise!
2 Views
Open in Slack
PreviousNext
Powered by