https://nf-co.re logo
Join Slack
Powered by
Thoughts on how to handle the CheckM2 issue? For c...
# mag
d
Thoughts on how to handle the CheckM2 issue? For context: we’re seeing a 403 error when downloading the CheckM2 database, apparently because Zenodo blocks requests with an 
aria2
user agent. We currently use aria2 for multi-stream downloads, since Zenodo transfers can be slow or unreliable. The simplest fix would be to change the user agent to bypass the block, though that might be considered a TOS violation. I couldn’t find any explicit mention of aria2 being disallowed, so I'm infering that from the block itself. Another option is to update the module to use 
wget
instead. A third option would be to deprecate the 
aria2/database_download
module and replace it with one based on https://github.com/dvolgyes/zenodo_get, which should be more robust than plain wget. This new module would be just 
zenodo_get
.
j
Is zenodo_get in bioconda/conda-forge?
d
j
In which case that would be my preference, as it puts the onus for success on an external tool rather than our roll-your-own checkm2 downloading module.
What was the reason that we can't just use the 
checkm2 database --download
command?
d
It's because using that option updates a file in the package itself to let checkm2 that the database is ready and where it is. That's possible on conda, but not on containers as they are read-only. And if it fails to update that file, throws an error.
👍 1
shakefist 1
j
The simplest fix would be to change the user agent to bypass the block, though that might be considered a TOS violation. I couldn’t find any explicit mention of aria2 being disallowed, so I'm infering that from the block.
I think we could probably advise this as a workaround for existing releases, but I agree I'd be wary about making a general change to the pipeline to incorporate this
And if it fails to update that file, throws an error.
Ridiculous
d
We can agree that, from a software design viewpoint, CheckM2 is somewhat... unconventional.
😇 1
😆 1
j
I agree with Jim
Let's go with teh dedicated tool
Hopefully they have better checks for fialures etc
d
nvm, Zenodo seems to be blocking zenodo_get too 🙄
j
curl
module with optional md5 check?
Though FWIW Nextflow will handle the raw Zenodo path:
Copy code
workflow {
    ch_input = channel.fromPath("<https://zenodo.org/api/records/5571251/files/checkm2_database.tar.gz/content>")


	UNTAR(ch_input)
}

process UNTAR {

    input:
	path tar

	output:
	path("**.dmnd")

	script:
	"""
	tar -xvzf ${tar}
	"""
}
Copy code
Staging foreign file: <https://zenodo.org/api/records/5571251/files/checkm2_database.tar.gz/content>
d
Yes, I guess we should do that now
Open in Slack
PreviousNext
Powered by