https://nf-co.re logo
Join Slack
Powered by
Since <@U07FNBFRQCE> started downloading the datab...
# proteinfold
f
Since @Keiran Rowell started downloading the databases, I wanted to finish this for their team. I see there are some folders in 
/mnt/shared/proteinfold_dbs
but it's only 300G, was the download interrupted? Does anyone know what the status of this is? otherwise, I can try to restart and download the databases. Do we have some code for downloading databses outside of the pipeline?
j
I don’t know whether someone tried to download the databases but I can point you to the dbs of all the old modes that we host in s3:
Copy code
# Test db for alphafold2
<s3://proteinfold-dataset/test-data/db/alphafold_mini>
# Test db for colabfold
<s3://proteinfold-dataset/test-data/db/colabfold_mini>
# Full db for alphafold2
<s3://proteinfold-dataset/db/alphafold/>
# Full db for colabfold 
<s3://proteinfold-dataset/db/colabfold>
# Full db for esmfold
<s3://proteinfold-dataset/db/esmfold>
👀 1
BTW, how can I get access to the gpu environment?
f
Send me a ssh public key and I will create an account for you!
There is a detailed walk-through and video in #C07PPQR415W
j
Some of the dbs are actually the same that are used by the new modes
The best would be to symlink the shared DBs and only download the DBs that are unique for each mode, this is how I have it on CRG HPC. I have some local scripts for downloading the DBs that are not in s3 links above but I will need to clean them a little bit and I have not time now, if guys in APAC or you @Florian Wuennemann can do it, I can do it tomorrow morning (CET)
f
@Jose Espinosa-Carrasco my plan was to aws s3 cp the dbs you listed above. Are there other things I should download?
j
yes, for the new modes in 
dev
(
helixfold3
, 
rosettafold_all_atom
, 
alphafold3
and 
rosettafold2na
)
for 
alphafold3
only the dbs can be shared as the parameters can not be used for commercial use and should be ask individually
and for rosettafold2na some formatting needs to be done
and as I said some of the DBs are actually used by different modes so that is why we should create symlinks for those
f
Ok I am still a bit confused. How do we get the dbs for the new modes?
j
You can find the links to download the dbs here: https://github.com/nf-core/proteinfold/blob/dev/conf/dbs.config
for each mode we have the *_path parameters to provide the paths if they have been already downloaded and the *_links from where the original dbs are downloaded
f
@Jose Espinosa-Carrasco The AWS S3 bucket is not public I am guessing... ?!
j
Yes, it's public
f
I wasn't able to copy out files with aws s3 cp earlier as it was complaining about credentials.
Sorry my bad, needed to add 
--no-sign-request
Works now 👍
How to fetch all the databases mentioned above on the cluster (I have already done this in 
/mnt/shared/proteinfold_dbs/
):
Copy code
# Test db for alphafold2
aws s3 cp <s3://proteinfold-dataset/test-data/db/alphafold_mini> ./alphafold_mini --recursive --no-sign-request

# Test db for colabfold
aws s3 cp <s3://proteinfold-dataset/test-data/db/colabfold_mini> ./colabfold_mini --recursive --no-sign-request

# Full db for alphafold2
aws s3 cp <s3://proteinfold-dataset/db/alphafold/> ./alphafold --recursive --no-sign-request

# Full db for colabfold 
aws s3 cp <s3://proteinfold-dataset/db/colabfold> ./colabfold --recursive --no-sign-request

# Full db for esmfold
aws s3 cp <s3://proteinfold-dataset/db/esmfold> ./esmfold --recursive --no-sign-request
🙌 1
j
What do you mean the rest of the dbs not in s3?
f
The ones for the new modes are not on S3 in your public bucket yet right?
j
No, they are not, sorry I missed this message
Open in Slack
PreviousNext
Powered by