Hi, I'd like to map metatranscriptomes to GTDB using magmap. How do I specify this in the

--genomeinfo

? Can I point it directly to GTDB?

No, one should use

--indexes

to

.sbt

files that the Sourmash people prepare: https://sourmash.readthedocs.io/en/latest/databases.html and turn on Sourmash search:

--sourmash

. To use kmer size 31:

--indexes <https://farm.cse.ucdavis.edu/~ctbrown/sourmash-db/gtdb-rs214/gtdb-rs214-reps.k31.sbt.zip> --sourmash --ksize 31

.

I saw it, and preparing the

params.yaml

now. I will start with one sample and see how it goes.

Every time you run, genomes will be selected based on the samples you have. If you run the pipeline multiple times for the same project with different sample sets, you will hence get different genome sets that you map against. My feeling is that this shouldn't matter but it might make your analysis less reproducible and somewhat more difficult to describe in the M&M and I would recommend a single run with all samples. I understand you're making a test run now, but think about this later.

Thank you for the information. My

ncbi_genome_infos.csv

contains:

./assembly_summary_genbank.txt

./assembly_summary_refseq.txt

I think you should keep the default for this. That just downloads the most recent files from NCBI. (Looks like this:

$ cat assets/ncbi_genome_infos.csv 
<https://ftp.ncbi.nlm.nih.gov/genomes/ASSEMBLY_REPORTS/assembly_summary_refseq.txt>
<https://ftp.ncbi.nlm.nih.gov/genomes/ASSEMBLY_REPORTS/assembly_summary_genbank.txt>

)

Ok. Initially I ran the pipeline with the following commands in the

params.yaml

:

indexes: './sourmash/gtdb-rs214-k31.sbt.zip'

sourmash: true

ksize: 31

Not surprisingly, I got an error regarding the

ncbi_genome_infos.csv

as this file, nor the directory

assets

existed. Hence, I downloaded the reference files manually and pointed to them in the

ncbi_genome_infos.csv

:

input: 'samplesheet.csv'

outdir: 'output'

indexes: './sourmash/gtdb-rs214-k31.sbt.zip'

sourmash: true

ksize: 31

ncbi_genome_infos: './ncbi_genome_infos.csv'

Normally, I'd just leave out the

ncbi_genome_infos:

line to let the pipeline use its default.

assets

is part of the pipeline code, so that should work without you doing anything. But the above should also work if you download the NCBI files to the paths you show above.

In the initial run I did leave out

ncbi_genome_infos

but it stopped with the error that

./assets/ncbi_genome_infos.csv

did not exist in the current wd.

Oh, good to know, we need to find a better default for that. Would you mind making an issue at the GitHub repo?

My magmap run stops at

SOURMASH_GATHER

and produced the following error:

Caused by:
  Process `NFCORE_MAGMAP:MAGMAP:SOURMASH:SOURMASH_GATHER (samples_sig)` terminated with an error exit status (1)


Command executed:

  sourmash gather \
      --threshold-bp "50" \
      --output samples_sig.csv.gz \
      --output-unassigned samples_sig_unassigned.sig.zip \
      --save-matches samples_sig_matches.sig.zip \
      --save-prefetch samples_sig_prefetch.sig.zip \
       \
      P33957_166_S243.sig P33957_165_S242.sig P33957_163_S240.sig P33957_156_S233.sig P33957_158_S235.sig P33957_155_S232.sig P33957_159_S236.sig P33957_161_S238.sig P33957_160_S237.sig P33957_164_S241.sig P33957_157_S234.sig P33957_162_S239.sig \
      gtdb-rs214-k31.sbt.zip
  
  cat <<-END_VERSIONS > versions.yml
  "NFCORE_MAGMAP:MAGMAP:SOURMASH:SOURMASH_GATHER":
      sourmash: $(echo $(sourmash --version 2>&1) | sed 's/^sourmash //' )
  END_VERSIONS

Command exit status:
  1

Command output:
  (empty)

Command error:
  loading from 'P33957_160_S237.sig'...
  
  loading from 'P33957_164_S241.sig'...
  
  loading from 'P33957_157_S234.sig'...
  
  loading from 'P33957_162_S239.sig'...
  
  loading from 'gtdb-rs214-k31.sbt.zip'...
  Traceback (most recent call last):
    File "/usr/local/bin/sourmash", line 11, in <module>
      sys.exit(main())
               ~~~~^^
    File "/usr/local/lib/python3.13/site-packages/sourmash/__main__.py", line 20, in main
      retval = mainmethod(args)
    File "/usr/local/lib/python3.13/site-packages/sourmash/cli/gather.py", line 203, in main
      return sourmash.commands.gather(args)
             ~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^
    File "/usr/local/lib/python3.13/site-packages/sourmash/commands.py", line 873, in gather
      databases = sourmash_args.load_dbs_and_sigs(
          args.databases,
      ...<5 lines>...
          fail_on_empty_database=args.fail_on_empty_database,
      )
    File "/usr/local/lib/python3.13/site-packages/sourmash/sourmash_args.py", line 335, in load_dbs_and_sigs
      db = _load_database(filename, False, cache_size=cache_size)
    File "/usr/local/lib/python3.13/site-packages/sourmash/save_load.py", line 117, in _load_database
      db = load_fn(
          filename, traverse_yield_all=traverse_yield_all, cache_size=cache_size
      )
    File "/usr/local/lib/python3.13/site-packages/sourmash/save_load.py", line 198, in _load_sbt
      db = load_sbt_index(filename, cache_size=cache_size)
    File "/usr/local/lib/python3.13/site-packages/sourmash/sbtmh.py", line 10, in load_sbt_index
      return SBT.load(
             ~~~~~~~~^
          filename,
          ^^^^^^^^^
      ...<2 lines>...
          cache_size=cache_size,
          ^^^^^^^^^^^^^^^^^^^^^^
      )
      ^
    File "/usr/local/lib/python3.13/site-packages/sourmash/sbt.py", line 857, in load
      storage = ZipStorage(location)
    File "/usr/local/lib/python3.13/site-packages/sourmash/sbt_storage.py", line 105, in __init__
      self._objptr = rustcall(lib.zipstorage_new, to_bytes(path), len(path))
                     ~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/usr/local/lib/python3.13/site-packages/sourmash/utils.py", line 78, in rustcall
      raise exc
  sourmash.exceptions.ZipError: Archive prepended with 8000 unknown bytes

Work dir:
  /cfs/klemming/projects/supr/naiss2025-23-231/distem/work/63/f839c44478d7cdbbb7cffc4dd3e4e8

Container:
  /cfs/klemming/projects/supr/naiss2025-23-231/singularity/depot.galaxyproject.org-singularity-sourmash-4.8.14--hdfd78af_0.img

Tip: when you have fixed the problem you can continue the execution adding the option `-resume` to the run command line
Jun-04 17:17:30.748 [TaskFinalizer-1] INFO  nextflow.Session - Execution cancelled -- Finishing pending tasks before exit
Jun-04 17:17:30.777 [Actor Thread 240] DEBUG nextflow.sort.BigSort - Sort completed -- entries: 5; slices: 1; internal sort time: 0.021 s; external sort time: 0.001 s; total time: 0.022 s
Jun-04 17:17:30.779 [Actor Thread 240] DEBUG nextflow.file.FileCollector - >> temp file exists? true
Jun-04 17:17:30.802 [Actor Thread 240] DEBUG nextflow.file.FileCollector - Retrieved cached collect-files from: /cfs/klemming/projects/supr/naiss2025-23-231/distem/work/collect-file/5916dd7fb779eb95a4964b315b074a42
Jun-04 17:17:30.804 [Actor Thread 240] DEBUG nextflow.file.FileCollector - Deleting file collector temp dir: /tmp/nxf-3575851337438453653
Jun-04 17:17:30.839 [Actor Thread 240] INFO  nextflow.container.SingularityCache - Pulling Singularity image <https://depot.galaxyproject.org/singularity/multiqc:1.28--pyhdfd78af_0> [cache /cfs/klemming/projects/supr/naiss2025-23-231/singularity/depot.galaxyproject.org-singularity-multiqc-1.28--pyhdfd78af_0.img]
Jun-04 17:17:31.283 [TaskFinalizer-1] ERROR nextflow.Nextflow - Pipeline failed. Please refer to troubleshooting docs: <https://nf-co.re/docs/usage/troubleshooting>
Jun-04 17:17:31.284 [Task monitor] DEBUG n.processor.TaskPollingMonitor - <<< barrier arrives (monitor: slurm) - terminating tasks monitor poll loop
Jun-04 17:17:31.285 [main] DEBUG nextflow.Session - Session await > all processes finished
Jun-04 17:17:31.285 [main] DEBUG nextflow.Session - Session await > all barriers passed
Jun-04 17:17:31.290 [Actor Thread 218] DEBUG nextflow.processor.TaskProcessor - Handling unexpected condition for
  task: name=NFCORE_MAGMAP:MAGMAP:MULTIQC; work-dir=null
  error [java.lang.InterruptedException]: java.lang.InterruptedException
Jun-04 17:17:31.621 [main] DEBUG nextflow.util.ThreadPoolManager - Thread pool 'TaskFinalizer' shutdown completed (hard=false)
Jun-04 17:17:31.622 [main] DEBUG nextflow.util.ThreadPoolManager - Thread pool 'PublishDir' shutdown completed (hard=false)
Jun-04 17:17:31.646 [main] INFO  nextflow.Nextflow - -[nf-core/magmap] Pipeline completed with errors-
Jun-04 17:17:31.724 [main] DEBUG n.trace.WorkflowStatsObserver - Workflow completed > WorkflowStats[succeededCount=60; failedCount=1; ignoredCount=0; cachedCount=0; pendingCount=0; submittedCount=0; runningCount=0; retriesCount=0; abortedCount=0; succeedDuration=3d 3h 30m 46s; failedDuration=15s; cachedDuration=0ms;loadCpus=0; loadMemory=0; peakRunning=20; peakCpus=144; peakMemory=864 GB; ]
Jun-04 17:17:31.724 [main] DEBUG nextflow.trace.TraceFileObserver - Workflow completed -- saving trace file
Jun-04 17:17:31.730 [main] DEBUG nextflow.trace.ReportObserver - Workflow completed -- rendering execution report
Jun-04 17:17:32.860 [Actor Thread 240] DEBUG nextflow.container.SingularityCache - Singularity pull complete image=<https://depot.galaxyproject.org/singularity/multiqc:1.28--pyhdfd78af_0> path=/cfs/klemming/projects/supr/naiss2025-23-231/singularity/depot.galaxyproject.org-singularity-multiqc-1.28--pyhdfd78af_0.img
Jun-04 17:17:32.866 [Actor Thread 221] DEBUG nextflow.util.HashBuilder - Hash asset file sha-256: /cfs/klemming/home/g/georgewe/.nextflow/assets/nf-core/magmap/assets/multiqc_config.yml
Jun-04 17:17:33.133 [main] DEBUG nextflow.trace.TimelineObserver - Workflow completed -- rendering execution timeline
Jun-04 17:17:33.642 [main] DEBUG nextflow.cache.CacheDB - Closing CacheDB done
Jun-04 17:17:33.694 [main] INFO  org.pf4j.AbstractPluginManager - Stop plugin 'nf-schema@2.2.0'
Jun-04 17:17:33.694 [main] DEBUG nextflow.plugin.BasePlugin - Plugin stopped nf-schema
Jun-04 17:17:33.695 [main] DEBUG nextflow.util.ThreadPoolManager - Thread pool 'FileTransfer' shutdown completed (hard=false)
Jun-04 17:17:33.696 [main] DEBUG nextflow.script.ScriptRunner - > Execution complete -- Goodbye

Any suggestion on how to proceed?

Which index file did you use?

The

gtdb-rs214-k31.sbt.zip

@Danilo Di Leo is this what you used for Envision?

I downloaded it. I could try

indexes: <https://farm.cse.ucdavis.edu/~ctbrown/sourmash-db/gtdb-rs214/gtdb-rs214-reps.k31.sbt.zip>

but it's the same file

Makes sense to download it since it's going to be fetched multiple times otherwise.

Navigating the website (https://sourmash.readthedocs.io/en/latest/databases.html) seems that r214 is the latest

A bit more googling found this: https://farm.cse.ucdavis.edu/~ctbrown/sourmash-db/gtdb-rs226/, i.e. the current release. Might be worth trying.

Ok, I'll give it a go. Thanks

Regarding the download: The file might have become corrupt during download, so downloading again might be worthwhile. But if the rs226 file works, you won't have to worry of course.

I checked the wget log and it looks fine. I downloaded it again and the number of bytes exactly matched. But I'll try the new one

It could well be that one of your sample

.sig.zip

files is what's causing the problem and not the index.

It seemed to be the index file (it just completed

SOURMASH_GATHER

). I downloaded the index file and updated the

params.yaml

:

indexes: './sourmash/gtdb-rs226-reps.k31.sig.zip'

Thanks for letting me know. And, nice side effect that I found the new index! 🙂