A new request to upload a dataset to nf-core AWS S3 bucket has been made! • Requested by: @Nick Waters • Pipeline name: funcprofiler • Why is this needed?: Running full size tests. - Humann v3 - fmhfunprofiler • Size of the dataset: large • URL: https://g-227ca.190ebd.75bc.data.globus.org/humann_data/uniprot/uniref_annotated/uniref90_annotated_v201901b_full.tar.gz https://g-227ca.190ebd.75bc.data.globus.org/humann_data/chocophlan/full_chocophlan.v296_201901b.tar.gz https://g-227ca.190ebd.75bc.data.globus.org/humann_data/full_mapping_v201901b.tar.gz http://cmprod1.cibio.unitn.it/biobakery4/metaphlan_databases/mpa_vJan25_CHOCOPhlAnSGB_202503.tar https://zenodo.org/records/10045253/files/KOs_sketched_scaled_1000.sig.zip • Related files: Core team will add this dataset to the nf-core bucket. Please react to the message with ✅ when this is done. [This process is not automated]

@Nick Waters what are the six of the different files approx?

Do the files need to be extracted?

Also that

uniref90 file is 20.5GB full_chocophlan 17.2GB full_mapping 2.7GB mpa is 4.8GB KOs ~ 100mb re extraction what do you normally do? Is the intent that these are compressed so people can save bandwidth running

test_full

, or just that these get used for tests run on aws, in which case they would need decompression?

For decompression, it depends on the pipeline - would your pipeline decompress them automatically? But otherwise for AWS it should not be a problem to upload the directories because nextflow/S3 it can pull those that for you (just doesn't work with HTTP). Having them uncompressed would make it faster to run the tests and cheaper

Yes, the pipeline models the database behavior in taxprofiler so dbs can be provided as either links or local paths. Ok so all these can be decompressed then, for easier AWS running. So it sounds like the expectation is that the bucket is primarily for running the tests on AWS, not as a download source for the users?

Mainly for running the tests

I've started downloading onto a cluster for upload 👍

Oh, sorry — does the data I’ve requested here meet the criteria from this convo? OR should these remain as direct links? the http one for sure needs to be uploaded, but should we hold off on the globus ones for now?

It's all fine!

Ok, thank you!!

Ok unpacked, getting ready for s3 upload

Will do! I’ve got some time blocked off for this tomorrow morning. thanks!