Hi all, I had requested a database upload to the s3 bucket, but forgot that a command needs to be run to format and index the database before it can be run by the tool using it. should I: a) somehow get shell access to an aws compute environment where I could log in and run the commands to format the database? b) request a re-upload from someone over at request-core, asking them to do the formatting command for me? c) adjust the pipeline to conditionally format the raw database if thats all thats provided? I'd rather not do c, if it can be avoided 🙂

I would suggest 2. Can it be formatted prior to asking for the re-upload though? ie can you format it first and we just upload the correctly formatted data

Sure, let me see if I can get a sharable Box link for the results, and then post that with the request to request-code. Unless you know of a better way to share the resulting files? they will be in the gb size range. Thanks for the input!

cc rest of <!subteam^S016E36N7TQ>

👉🏻 #C09H6NYHR9T

Its for metaphlan, a different version than that used by taxprofiler. I don't think there is a pre-formatted copy of it publicly available. Should I just skip that tools full test? Or is there a way to specify "don't use this for production runs, this config is just for aws tests"?

If there's a version publcily available then can the config just point directly to that instead?

I don't think there is a pre-formatted copy of it publicly available.

Isn't there a preprocessing module available that can be run as ‘setup’ in nf-test?

@Jim Downie has a trick for that, but MP databases might be too hardcore

How big is the DB? Are we talking MBs, GBs, TBs?

GBs

roughly how many? 10s or 100s?

~34GB

The processing step is basically running bowtiew in the background too

34GB isn't too bad, but depends a bit on how popular the pipeline is 😅

So it sounds like the consensus then is to skip aws megatesting on this tool? • There isn't a good public download of the indexed version that can be pulled directly • The public download of the non-indexed version is slow • index it during pipeline runs is wasteful of time and compute • hosting an indexed version on aws is similarly wasteful of storage, and would incur high egress costs if abused

Since it's reference data, is igenomes an option for an indexed version?

It's marker regions of reference genomes, not whole genomes -- would that pose an issue?

Wait, what version of the database @Nick Waters? And is it just MP or full human?

for metaphlan; the version of humann 3 we are supporting uses a metaphlan 4 database; the taxprofiler one as of cd7ef63 is labeled metaphlan 3

Also: I'm mostly worried about people running the pipeline. I'm slightly less worried about only-megatests. Though folks do sometimes run benchmarking efforts with megatests / full-size test configs, which can pull a lot of data quickly.

Before converting this pipeline to nextflow our solution was a separate database download/building/indexing workflow. We can point users to that as a solution (and have the workflow generate a

databases.csv

file at the end, which could be used for nf-core/funcprofiler input). Its not perfect by any stretch, but I think anyone familiar with functional profiling knows they are signing up for some baseline level of database wrangling