Hi, New to slack and also to taxprofiler to which I try to send this question I am trying to prepare taxprofiler to run on my server and am struggling building the multiple required databases I will be analyzing metagenomic shotgun data that will contain bacteria and archaea but maybe also fungi, host DNA, and other species. Which reference databases should I build for taxprofiler? (for each of the many tools there) Do I collect only bacteria and archaea or do I also include other refseq fasta references (the full NR?)? I was pointed to the dev tool https://nf-co.re/createtaxdb/dev but there nothing really specific is said about the source of reference genus to put in the config file In my case this is unknown and I want a pipeline reusable for other experiments and do not work with the same ecosystems each time. Thanks for your feedback #C031QH57DSS

Hi @Stephane Plaisance! Welcome to the slack! Unfortunately there are no standard databases really. The closest you have is Ben langmead's AWS index zone thing. They are probably OK for general screening, but ultimately your database design will depend on your actual question. I don't fully understand your question about #C062H82UCRG in regards to the reference genus - could you expand in more detail what you don't understand?

Hi James, I need to use taxprofiler to analyze shotgun sequencing from various projects, some experiments will have human or mouse as host (fecal extracts), other will come from plant hosts (symbiotes in the roots), others will come from soil or whatever ecological location studied in that project. So I need to have the right reference data prepared inorder to deliver the right results. My question is which genomes should be assembles in the various databases used by the pipeline? (centrifuge, diamond, ...) I did not understand (yet) what genomes need to be present in the many tool databases used in the pipeline and what each tool is able to look for besides bacteria. Of course we are interested mainly by the bacteria and archaea BUT knowing which other genomes are present (eg fungi or even viruses) can be key to a project. Sorry if my question is naive, I never analyzed shotgun yet!

Hi Stephane, if I were you I would start as simple as possible. And that is to use a pre-prepared database. I am most familiar with Kraken2, so I'd recommend that. You can choose a database from https://benlangmead.github.io/aws-indexes/k2. For example, if it shall include basically everything, use the database listed as "core_nt Database". Pre-download the ".tar.gz" and list it as described in https://nf-co.re/taxprofiler/1.2.4/docs/usage/#full-database-sheet. If thats your first attempt, you might want to start with a smaller database though. Having said that, James is the expert with the pipeline. Just passing by 😉

Hey @Stephane Plaisance As Daniel says, generally if you 0 clue where to start, a generic database like Ben Langmead ones I mentioned above, and Daniel just linked, is probably your best bet. They are very commonly used for general screening to get a rough idea of what are in your samples. But really when it comes to metagenomics, like I said - it will depend on what you are looking for and the ecosystem you are looking at... At it's core, all you need for #C062H82UCRG (which we should move over there first) is reference FASTA files of the genomes oyu are interested in. Nothing more. What I have done in the past is go to the NCBI RefSeq FTP server, download the assembly reports of the kingdoms you're interested in then do various filters for assembly quality, and diversity to target what you are interested in. In some cases you may need to make the the number of genomes smaller to fit into your computing resource, so you could do random picking to select one reference genome per species, or remove entire genera or families of things you know absolutely are not in your ecosystem. Once you've filtered those tables, one of the columns gives URLs to the genome FASTA files on the NCBI FTP servers and can download them from there. If you are interested in more bacterial diversity that includes MAGs, you can do the same thing roughly with the GTDB database too. Then you basically can do the same thing on the NCBI servers to select the reference genome you need to remove host reads prior profiling in #C031QH57DSS (just in this case it'll be a single genome, and you're going for the highest quality and most complete reference genome) Those are the technical steps and hopefully can give you a rough idea of the procedure on how to get your genomes - but we can't really give you advice on your study design (which I think is maybe what you are looking for? It's still not clear to me exactly, sorry) ... for that you need to talk to your project partners... we can help you with getting the pipelines set up and running but as volunteers we can't really give you generalised scientific advice.

thanks a lot @James Fellows Yates for the detailed answer I am OK with using the Langmead downloads for a start, only thing is that they represent a couple of the many DB listed inn the install section Can I run without all the others DBs which are so poorly described (or which fasta should be fed to make them)? You write that I should include only what I need, how can I know what I need if my project can be anything with bacteria, archaea, viruses, fungi and host eukaryotes? I am creating with a lot of pain a centrifuge DB with prok+viruses+fungi+mouse (my host so far) but the centrifuge build command crashes each time due probably to RAM overflow (512G avail) and I do not get all the db files. And this is only centrifuge, which of the other DBs do I need to prepare? I made a general diamond DB with all NR fasta from ncbi. The taxprofiler install documentation describes the commands for all other DBs but not the FASTA sources except for simplistic examples with 5 fasta (not realistic!) The whole tutorial seems written by ITs who have no clue about what to put in the DBs or experts that presume users will know what to use. Sorry but this is very difficult for me (no offense meant here, just overwhelmed by the task - and I am not even starting to run taxprofiler)

Can I run without all the others DBs which are so poorly described

And this is only centrifuge, which of the other DBs do I need to prepare?

You do not have to run all the profiling tools! The pipeline is completely flexible. You only need to supply databases for the tools you want to run, and supply the corresponding pipeline parameter to activate that profiler. Have you seen our 'simple tutorial'? This demonstrates that: Have you seen the 'simple tutorial' here? https://nf-co.re/taxprofiler/1.2.4/docs/usage/usage/tutorials/ Otherwise if you're unsure which profiler to use, should read the literature in the area you are working in and select the profilers that work best for you. You can see the CAMI challenge papers that can maybe guide you to a certain extent which tool is best for which purpose (but it is often far from clear all have their pros and cons!)

You write that I should include only what I need, how can I know what I need if my project can be anything with bacteria, archaea, viruses, fungi and host eukaryotes?

Well it depends on the study design! How important is the quality of genomes? Do you want accuracy or sensitivity? How much RAM do you have? None of these we can advise you on as we are not involved in your study. As I said we can only provide 'technical' support for executing the pipeline.

The whole tutorial seems written by ITs who have no clue about what to put in the DBs or experts that presume users will know what to use.

Sorry but this is very difficult for me (no offense meant here, just overwhelmed by the task - and I am not even starting to run taxprofiler)

I understand and recognise metagenomics hard! It is by far from a standardised field, it does require a lot of testing and exploration still. But so far, we've had many novice users who have not had issues with setting up the pipeline. They did however have a basic/OK knowledge of the concepts of metagenomics. If you're struggling to navigate this, it may be worth finding collaborators with experience to help you make some of these decisions, based on your study design.

Thanks James for your patience and detailed comments. I fully agree of course with most of your points. Concerning the choice of DB to use, I tend to disagree, our scientists do fundamental research and often want to discover something unexpected or unknown rather than only perform classificqation of bacteria present in a sample. This can include for instance the discovery of an over-represented phage or of eukaryote genomes in what was believed to be a fully bacterial community. To achieve this we need more than the default databases (I guess) or at least the default but containing more than just bacteria species. I will do my best to implement as many as possible of the available databases and see where I end. Best regards,