Hey proteinfold team, Just doing some testing of nf-core/proteinfold on AWS and seeing really long runtimes for test profiles. Even though the databases point to

mini

versions, the files are quite large. Any guidance on decreasing runtime when running on AWS with S3?

I havent used the dbs from the profile but we use different mini dbs locally which should be a good amount faster. I think @Jose Espinosa-Carrasco was planning to add these new mini dbs to the S3 bucket.

Thanks for the insight @Thomas Litfin. When you say locally, I am guessing you are running on HPC with a local filesystem or do you mean you have mini dbs on something like FSx or another file system for AWS?

Sorry, I mean on our HPC local file system.

Ahh yes, for sure using DBs on a local file system will always be much better performance than on cloud unfortunately (except if you mount a local filesystem). My question regarding the mini dbs still remains. @Jose Espinosa-Carrasco are we sure the "mini" databases are actually mini? I checked and the mgnify db for the alphafold multimer test is 113GB alone...

Sure, a local filesystem is faster but the local dbs that I mentioned are different to the hosted copies. Eg our mgnify is a few MB.

https://drive.google.com/file/d/1XOfzXf-pFDGwiLN0Dv8lm6NDEfvO_mOG/view?usp=sharing These are the miniDBs I shared with Jose

Thanks for sharing @Joshua. The data that under : s3://proteinfold-dataset/test-data/db/alphafold_mini is definitely bigger than that. For example, the following file : s3://proteinfold-dataset/test-data/db/alphafold_mini/mgnify/mgy_clusters_2022_05.fa is 119.7 Gb alone. So I think there was a mixup in the upload to the mini databases potentially?!

Hi, some of the DBs on alphafold_mini currently we weren’t able to make then smaller, what @Joshua did with some of them is to take just the hits we needed for our testing, which is a hack good enough for our testing, not sure if this is the case of mgnify or he actually was able to shrink this one. But be aware that some of the DBs in the google drive might only contain the hits of our test data while with

alphafold_mini

you should be able to run any input sequence