https://nf-co.re logo
Join Slack
Powered by
A new request to upload a dataset to nf-core AWS S...
# request-core
u
A new request to upload a dataset to nf-core AWS S3 bucket has been made! • Requested by: @Max Cummins • Pipeline name: CheckM2 - Module • Why is this needed?: ChecKM2 - Zenodo seems to have changed their configuration settings, preventing chunk-parallelized downloads. Now only single threaded downloads work, but these take 15-30 minutes to complete for nf-tests. Smaller databases cannot be produced: See https://github.com/nf-core/mag/issues/915 Trying to fix the broken CheckM2 module: https://github.com/nf-core/modules/pull/9744#issuecomment-3823122732 • Size of the dataset: 16GB • URL: https://zenodo.org/records/14897628/files/checkm2_database.tar.gz • Related files: Core team will add this dataset to the nf-core bucket. Please react to the message with ✅ when this is done. [This process is not automated]
✅ 1
j
I'll take this over <!subteam^S016E36N7TQ>
🙏🏻 1
🙏 1
But need to think about where to put it, so might take a little
Hi @Max Cummins FYI it's only 1.6 GB not 16GB. But good news at least is that I had actually already uplaoded to the nf-core s3 bucket for #CE9MS66BS (even if we don't use it directly, as we don't want people to pull it outisde as AWS, and I know people like running 
test_full
on their cluster sometimes), but should be fine for modules: 
<s3://nf-core-awsmegatests/mag/full_test_data/CheckM2_database/14897628/uniref100.KO.1.dmnd>
r
Does this modules testdata need to be of this size
j
In this case yes, there is no way of making a smaller test database currently
(@Jim Downie is waiting on a PR to be merged that would allow this)
r
@Jonathan Manning fyi
j
(but won't be coming any time soon)
j
Thanks! Yep, came across this issue (https://zenodo.org/records/14897628/files/checkm2_database.tar.gz) via the Galah tests last week. 1.7Gb!
j
1.7GB should fit on the nodes though right? There are connection issues with Zenodo atm though as they've been throttling/blocking a lot of stuff because of AI crawlers
old man yells at claude 2
m
j
Oooooh that's new!
m
Thanks James - appreciate it 🙂
j
FWIW the Galah test should really just use the pre-calculated output of CheckM2 on the input genomes it receives. I think I was feeling rather avoidant of a test-datasets PR when I wrote it, which was really just setting us up for failure 🫢🙈
😆 1
m
Oh so is CheckM2 is not deterministic @Jim Downie? My Galah tests just seem to fail sometimes but not other times
➕ 1
This has been such a convoluted attempt at updating versions; zenodo rate limiting side quest and galah test final boss 😂
🫣 1
j
I would like to say it should be deterministic - the gene calling part certainly should be -but it runs an ML model so there may be randomness on the outputs sometimes...
m
yeah right
thats what I was thinking
j
I saw it too, we should definitely update the test not to expect static output
m
I get this quite often
➕ 1
and the tests take like 40 mins to run
by which time main branch has updated
and I need to update main branch and hope tests pass again
and they often dont because of rate limiting nonsense from zenodo or galah fail
haha
j
😭😭😭
I will run the Galah test and make a PR to test-datasets to upload the CheckM2 TSV
I will repent my past laziness...
m
haha
That would be helpful ❤️
thanks
j
shame
😇 1
j
(I have the database pre-downloaded so it hopefully shouldn't take long!)
🙌 1
i hate cloning test-datasets
🫠 1
m
Screenshot 2026-02-03 at 13.46.33.png
🤣
j
I'll fix it anyway
🙌 1
m
no i mean
branch out of date
j
Oh
🫠
m
let me know when you need a force merge 😉
🙌 1
m
Can you force merge if branch is out of date?
im kinda conflicted as the tests are inherently unstable
Jim is sorting one
the zenodo
is not stable
so force merge pushes it onto someone elses plate to some degree
m
good, just wanted to let you know that you can ignore the main branch for now
m
Great, thanks 🙂
m
thanks for trying to sort this out!
👍 1
j
m
nice Jim, thanks
j
I'll make a PR for the galah test now also
🙌 1
m
legend
I think I should use the CheckM2 database from AWS now to sidestep the zenodo rate limit issues That means this part of the code is no longer an option, which is used to parse a checksum
Copy code
def downloadZenodoApiEntry(zenodo_id) {
    // Download metadata from Zenodo API, setting "Accept: application/json" header
    def api_url  = "<https://zenodo.org/api/records/${zenodo_id}>"
    def conn     = new URL(api_url).openConnection()
    conn.setRequestProperty('Accept', 'application/json')
    conn.setRequestProperty('User-Agent', "Nextflow ${nextflow.version ?: ''}".trim())

    def api_text = conn.getInputStream().getText('UTF-8')
    def parser   = new groovy.json.JsonSlurper()

    return parser.parseText(api_text)
}
Should I just strip the functionality of matching the checksum?
any suggestions appreciated - I feel if i dont swap from zenodo to AWS now this just kicks the can down the road
j
I think I should use the CheckM2 database from AWS now to sidestep the zenodo rate limit issues
We can do but we need to be very careful because otherwise nf-core will 'pay' for it if people use the pull the files regularly outside of AWS
m
egress costs?
j
Exactly
m
right
well zenodo may stabilise according to the post shared above
so maybe for now we leave it as zenodo
j
It might take a while though
So if we are having lots of problems we could consider switching (I don't think checkmn2 is SUPER popular)
m
Well we have two options as far as I can see: 1. Strip out checksum functionality from the module and pull database from AWS rather than zenodo, and dealing with any egress issues downstream if they arise, potentially swapping back to zenodo 2. Merge current module which pulls from zenodo and retains checksum functionality but will sometimes cause tests to fail until zenodo stabilises their servers
👍 1
Screenshot 2026-02-03 at 14.06.00.png
cowboyhat tip 1
😬 1
this is official nf-core guidelines I think
😆 1
let me know when you need a force merge 😉
@mashehu
m
a-video-game-character-is-saying-i-have-been-summoned.gif
😂 1
j
j
this is official nf-core guidelines I think
if it's a sticker!
😆 1
m
its-done-finished.gif
🤣 1
m
Thanks all :')
🫂 2
2 Views
Open in Slack
PreviousNext
Powered by