A community effort to collect a curated set of analysis pipelines built using Nextflow.

nf-core

image.png

Is it reasonable for `--build_kraken2` on 7 eukaryotic genomes to take over 2 hours? I'd like to add all archaea and bacteria to this for host removal, but I must be doing something wrong...
`KRAKEN2_ADD` is not parallelizing but I guess that is normal for reading the ~14GB `nucl_gb.accession2taxid`?

As an aside, the `--build_kraken2` flag is giving me the following error
```ERROR ~ Validation of pipeline parameters failed!

 -- Check '.nextflow.log' file for details
The following invalid input values have been detected:

* --build_kraken2 (true): Value is [string] but should be [boolean]```
I can get it to run by providing this config file though
```params {
    build_kraken2 = true
}```
Ah figured out this issue at least: <https://nf-co.re/blog/2026/parameter-types|nf-co.re/blog/2026/parameter-types>

Hi <@U09GTGE7L92> yeah I actually tried to address that yesterday using the 'dev guide' that Nicolas kindly wrote, but unfortunately the state of the Nextflow implementation was suboptimal so I dropped it. I'll do a patch release soon with just bumping the nf-schema bersion

Oh actually that was already in dev and I forgot :facepalm: starting release proceudre now!

Thanks <@UEM37TBAR>
Any advice on the run time though?

That no unfortunately. Maybe you can look inside the corresponding `work/`


Directory to see what the `.command.log` says

```INFO:    Environment variable SINGULARITYENV_TMPDIR is set, but APPTAINERENV_TMPDIR is preferred
INFO:    Environment variable SINGULARITYENV_NXF_TASK_WORKDIR is set, but APPTAINERENV_NXF_TASK_WORKDIR is preferred
INFO:    Environment variable SINGULARITYENV_NXF_DEBUG is set, but APPTAINERENV_NXF_DEBUG is preferred
Creating sequence ID to taxonomy ID map (step 1)...
Found 8698/8698 targets, searched through 242044067 accession IDs, search complete.
Sequence ID to taxonomy ID map complete. [1m26.253s]
Estimating required capacity (step 2)...
Estimated hash table requirement: 18286732432 bytes
Capacity estimation complete. [3m1.943s]
Building database files (step 3)...
Taxonomy parsed and converted.
CHT created with 6 bits reserved for taxid.
Completed processing of 8698 sequences, 19404142574 bp
Writing data to disk...  complete.
Database files completed. [43m36.041s]
Database construction complete. [Total: 48m4.274s]
Database disk usage: 18G
After cleaning, database uses 18G```

Ah wait I just noticed you said eukaryotic genomes

Depending on the size that could indeed take a long time

Mostly rodents and a human genome. Maybe this is to be expected

Yes they are large, so that sounds normal too

When adding a lot of genomes, seems like most of the run time is dealing with HTTP 503 errors before `NFCORE_CREATETAXDB:CREATETAXDB:PREPROCESSING:UNPIGZ_DNA` . Is this throttling by NCBI?

```[main] DEBUG n.file.http.XFileSystemProvider - Got HTTP error=503 waiting for 250ms (attempt=1)```

Possibly, might be rate limiting because bots.

While nextflow _can_ do it I generally wouldn't recommend trying to give URLs directly in the samplesheet.

I would use the ncbi `datasets` tool to download it using their 'dehydrate' and `rehydrate` mechanism