I'm going to download some data using the new(ish)...
# modules
i
I'm going to download some data using the new(ish) NCBI Datasets CLI. I though about adding at least parts of it to nf-core. However, it has up to 5 levels of subcommands (cf. diagram below). Would it have to have 4/5 levels of directories to cover every subcommand?
👍 1
😱 3
m
Only 1 sublevel as far as I'm concerned
m
For the record, there’s an issue and a draft PR
🎉 1
j
If you add this as an nf-core module, it might be worth having a conversation about choosing a standard name for the NCBI API key as a Nextflow secret. I'm planning a change to another module (Mitohifi) sometime soon to add this, so it would be good to co-ordinate this across multiple modules so there aren't multiple secrets with similar but different names.
➕ 1
m
I think @Zachary Foster was thinking in a similar way
f
Only 1 sublevel as far as I'm concerned
but then check which subcommands make sense as their own modules and which can be swapped using val channels
this 1
so maybe datasets/download can be one module with value channels that then set whether a gene, a virus genome or taxonomy is downloaded
this 1
i
This setup puts quite a lot of responsibility on the user, but it should be fine
f
otherwise we have a lot of modules that essentially do the same thing right?
this 1
m
for me modules should be modular
j
I would personally go: DATASETS_DOWNLOADGENOME DATASETS_DOWNLOADGENE DATASETS_DOWNLOADVIRUS DATASETS_DOWNLOADTAXONOMY DATASETS_REHYDRYATE
i
It's a bit tricky because e.g.
summary taxonomy
and
summary gene
have different arguments, so they are not 100% compatible, although they have the same Nextflow structure as the difference lies in
ext.args
🫥 1
j
DATASETS_DOWNLOADGENOME DATASETS_DOWNLOADGENE DATASETS_DOWNLOADVIRUS DATASETS_DOWNLOADTAXONOMY DATASETS_SUMMARYTAXONOMY DATASETS_SUMMARYGENE DATASETS_REHYDRYATE
z
This would be useful! Our pipeline has a custom module to do `datasets download genome accession`: https://github.com/nf-core/pathogensurveillance/blob/dev/modules/local/download_assemblies/main.nf
Currently I am having issues with "Too many requests" if
maxForks
> 1 even with an API key set up using secrets, even though it used to work with
maxForks
<= 5 even without a key.