https://nf-co.re logo
Join Slack
Powered by
Hi all, happy Friday! whats the process for gettin...
# request-core
n
Hi all, happy Friday! whats the process for getting full databases deposited in s3://nf-core-awsmegatests?
p
Use one of the workflows in this channel (test datasets on s3) and fill in the form
n
Ok, I thought that was for test data not actual data. Will do!
p
Meh, it's for data
We'll ask you why you need it either way 😅
n
Thanks. I’m going to work an a PR addressing this issue — is there anything else newbies should know about completing this process?
p
It's one of these things that is a kind of slightly under-documented, because it's done on a case by case basis
We prefer not to upload anything, it's kind of our last-resort solution
So I guess the docs should just say "contact the core team to discuss on slack channel x"
n
Oh interesting, what is the best-case scenario then, in terms of locations for large datasets and databases? I think everything I assumed about this bucket has been incorrect so far
😅 1
p
We prefer to download directly from the authoritative source
eg. just put the URL in the pipeline code so that Nextflow fetches it at run time
If it's in our s3 bucket then we have to pay for hosting and egress fees
Which is fine - we have AWS credits so can cover it, but if ever our AWS credits run out or otherwise cease to be, the pipeline will fail
n
Gotcha. So this bucket is primarily for stuff locked behind a login page (eg JGI downloads) or otherwise not conveniently accessible? or as James mentioned in the other thread, stuff at public but insecure http links? I’m happy to note that in the PR if that would be helpful
p
Yeah basically. We started doing it for downloads that failed a lot etc.
Open in Slack
PreviousNext
Powered by