CheckM EOF error with

profile test,singularity

on Nextflow 24.04.4 and nfcore-mag v3.0.3 Hi, I am getting the following CheckM EOF error. I checked by logs and the system I'm running on is not running out of memory or disk space. Interestingly, under the

GenomeBinning/QC/CheckM/*/checkm.log

folder reports that checkm finished without errors. But I'm not sure why it's throwing the EOF error. Would anyone happen to know what I could change to fix this? Thank you!

nextflow run nf-core/mag -r 3.0.3 -profile test,singularity --outdir test_dir2 --skip_quast true --skip_clipping true --skip_prodigal --skip_prokka --skip_concoct --skip_gtdbtk true --binqc_tool checkm --checkm_db /projects/bacteriology_tran_data/MY_CHECKM_FOLDER --refine_bins_dastool false

The error message:

Command output:
  [2024-09-04 14:43:21] INFO: CheckM v1.2.1
  [2024-09-04 14:43:21] INFO: checkm lineage_wf -t 2 -f SPAdes-MaxBin2-unclassified-unrefined-test_minigut_wf.tsv --tab_table --pplacer_threads 2 -x fa input_bins/ SPAdes-MaxBin2-unclassified-unrefined-test_minigut_wf
  [2024-09-04 14:43:21] INFO: CheckM data: MY_CHECKM_FOLDER
  [2024-09-04 14:43:21] INFO: [CheckM - tree] Placing bins in reference genome tree.
  [2024-09-04 14:43:21] INFO: Identifying marker genes in 2 bins with 2 threads:
  
  Unexpected error: <class 'EOFError'>

Command error:
  INFO:    Converting SIF file to temporary sandbox...
  [2024-09-04 14:43:21] INFO: CheckM v1.2.1
  [2024-09-04 14:43:21] INFO: checkm lineage_wf -t 2 -f SPAdes-MaxBin2-unclassified-unrefined-test_minigut_wf.tsv --tab_table --pplacer_threads 2 -x fa input_bins/ SPAdes-MaxBin2-unclassified-unrefined-test_minigut_wf
  [2024-09-04 14:43:21] INFO: CheckM data: MY_CHECKM_FOLDER
  [2024-09-04 14:43:21] INFO: [CheckM - tree] Placing bins in reference genome tree.
  [2024-09-04 14:43:21] INFO: Identifying marker genes in 2 bins with 2 threads:
  Process SyncManager-1:
  Traceback (most recent call last):
    File "/usr/local/lib/python3.10/multiprocessing/process.py", line 315, in _bootstrap
      self.run()
    File "/usr/local/lib/python3.10/multiprocessing/process.py", line 108, in run
      self._target(*self._args, **self._kwargs)
    File "/usr/local/lib/python3.10/multiprocessing/managers.py", line 591, in _run_server
      server = cls._Server(registry, address, authkey, serializer)
    File "/usr/local/lib/python3.10/multiprocessing/managers.py", line 156, in __init__
      self.listener = Listener(address=address, backlog=16)
    File "/usr/local/lib/python3.10/multiprocessing/connection.py", line 453, in __init__
      self._listener = SocketListener(address, family, backlog)
    File "/usr/local/lib/python3.10/multiprocessing/connection.py", line 596, in __init__
      self._socket.bind(address)
  OSError: [Errno 98] Address already in use
  
  Unexpected error: <class 'EOFError'>
  Traceback (most recent call last):
    File "/usr/local/bin/checkm", line 856, in <module>
      checkmParser.parseOptions(args)
    File "/usr/local/lib/python3.10/site-packages/checkm/main.py", line 979, in parseOptions
      self.tree(options)
    File "/usr/local/lib/python3.10/site-packages/checkm/main.py", line 157, in tree
      binIdToModels = mgf.find(binFiles,
    File "/usr/local/lib/python3.10/site-packages/checkm/markerGeneFinder.py", line 67, in find
      binIdToModels = mp.Manager().dict()
    File "/usr/local/lib/python3.10/multiprocessing/context.py", line 57, in Manager
      m.start()
    File "/usr/local/lib/python3.10/multiprocessing/managers.py", line 566, in start
      self._address = reader.recv()
    File "/usr/local/lib/python3.10/multiprocessing/connection.py", line 255, in recv
      buf = self._recv_bytes()
    File "/usr/local/lib/python3.10/multiprocessing/connection.py", line 419, in _recv_bytes
      buf = self._recv(4)
    File "/usr/local/lib/python3.10/multiprocessing/connection.py", line 388, in _recv
      raise EOFError
  EOFError
  INFO:    Cleaning up image...

Work dir:
  /var/lib/condor/execute/slot1/dir_731726/work/d8/1f66bf620d85798cc9c9ea6091277b

Tip: when you have fixed the problem you can continue the execution adding the option `-resume` to the run command line

 -- Check '.nextflow.log' file for details
ERROR ~ Pipeline failed. Please refer to troubleshooting docs: <https://nf-co.re/docs/usage/troubleshooting>

 -- Check '.nextflow.log' file for details

Example of a

checkm.log

under the results folder:

[2024-09-04 14:43:19] INFO: CheckM v1.2.1
[2024-09-04 14:43:19] INFO: checkm lineage_wf -t 2 -f SPAdes-MetaBAT2-unclassified-unrefined_unbinned-test_minigut_wf.tsv --tab_table --pplacer_threads 2 -x fa input_bins/ SPAdes-MetaBAT2-unclassified-unrefined_unbinned-test_minigut_wf
[2024-09-04 14:43:19] INFO: CheckM data: MY_CHECKM_FOLDER
[2024-09-04 14:43:19] INFO: [CheckM - tree] Placing bins in reference genome tree.
[2024-09-04 14:43:19] INFO: Identifying marker genes in 2 bins with 2 threads:
[2024-09-04 14:43:20] INFO: Saving HMM info to file.
[2024-09-04 14:43:20] INFO: Calculating genome statistics for 2 bins with 2 threads:
[2024-09-04 14:43:20] INFO: Extracting marker genes to align.
[2024-09-04 14:43:20] INFO: Parsing HMM hits to marker genes:
[2024-09-04 14:43:20] INFO: Extracting 43 HMMs with 2 threads:
[2024-09-04 14:43:20] INFO: Aligning 43 marker genes with 2 threads:
[2024-09-04 14:43:20] INFO: Reading marker alignment files.
[2024-09-04 14:43:20] INFO: Concatenating alignments.
[2024-09-04 14:43:20] INFO: No genomes were identified that could be placed in the reference genome tree.
[2024-09-04 14:43:21] INFO: { Current stage: 0:00:01.675 || Total: 0:00:01.675 }
[2024-09-04 14:43:21] INFO: [CheckM - lineage_set] Inferring lineage-specific marker sets.
[2024-09-04 14:43:21] INFO: Reading HMM info from file.
[2024-09-04 14:43:21] INFO: Parsing HMM hits to marker genes:
[2024-09-04 14:43:21] INFO: Determining marker sets for each genome bin.
[2024-09-04 14:43:22] INFO: Marker set written to: SPAdes-MetaBAT2-unclassified-unrefined_unbinned-test_minigut_wf/lineage.ms
[2024-09-04 14:43:22] INFO: { Current stage: 0:00:01.138 || Total: 0:00:02.813 }
[2024-09-04 14:43:22] INFO: [CheckM - analyze] Identifying marker genes in bins.
[2024-09-04 14:43:22] INFO: Identifying marker genes in 2 bins with 2 threads:
[2024-09-04 14:43:32] INFO: Saving HMM info to file.
[2024-09-04 14:43:32] INFO: { Current stage: 0:00:10.542 || Total: 0:00:13.356 }
[2024-09-04 14:43:32] INFO: Parsing HMM hits to marker genes:
[2024-09-04 14:43:32] INFO: Aligning marker genes with multiple hits in a single bin:
[2024-09-04 14:43:32] INFO: { Current stage: 0:00:00.086 || Total: 0:00:13.442 }
[2024-09-04 14:43:32] INFO: Calculating genome statistics for 2 bins with 2 threads:
[2024-09-04 14:43:32] INFO: { Current stage: 0:00:00.022 || Total: 0:00:13.465 }
[2024-09-04 14:43:32] INFO: [CheckM - qa] Tabulating genome statistics.
[2024-09-04 14:43:32] INFO: Calculating AAI between multi-copy marker genes.
[2024-09-04 14:43:32] INFO: Reading HMM info from file.
[2024-09-04 14:43:32] INFO: Parsing HMM hits to marker genes:
[2024-09-04 14:43:32] INFO: QA information written to: SPAdes-MetaBAT2-unclassified-unrefined_unbinned-test_minigut_wf.tsv
[2024-09-04 14:43:32] INFO: { Current stage: 0:00:00.069 || Total: 0:00:13.534 }

Hi there, as far as I understand that is an error likely on the side of your OS:

OSError: [Errno 98] Address already in use

resume the pipeline (append

-resume

) and it should work.

Thank you for the reply Daniel. Unfortunately, I'm not able to use the

resume

option because the way I'm running jobs is through the HTC job manager, so when the job fails/finishes, all the intermediary files are lost

Not having resume available is quite uncinvenient, maybe speak with your sys admin. If you do not have resume working, than simply start the pipeline a few times and see how far the pipeline comes. The idea here is to figure out whether that is a problem with your data/pipeline (fails always at the same point) or an OS problem (fails at different points or even succeeds occationally).

Good idea thanks. I'll submit 10 iterations of the job using the same settings and check the logs to see if it's a problem with the pipeline or the OS. Thanks again.