Hey, all! I did a run with ampliseq a while back and need to redo the pipeline with an updated metadata sheet. Unfortunately, I'll have to kick off the pipeline again since I already removed the work folder. I put the new metadata sheet in the folder with the script, fastq files, and all the configuration documents. I made sure to specify the version of both Nextflow and of ampliseq that was used in the first run. This time, I'm getting the following error, which I did not get before:

ERROR ~ Error executing process > 'NFCORE_AMPLISEQ:AMPLISEQ:FORMAT_TAXONOMY'

Caused by:
  Can't stage file <https://figshare.scilifelab.se/ndownloader/files/47244061> -- reason: Unable to access path: <https://figshare.scilifelab.se/ndownloader/files/47244061>



Container:
  docker.io/biocontainers/biocontainers:v1.2.0_cv1

Tip: you can replicate the issue by changing to the process work dir and entering the command `bash .command.run`

 -- Check '.nextflow.log' file for details
ERROR ~ Pipeline failed. Please refer to troubleshooting docs: <https://nf-co.re/docs/usage/troubleshooting>

 -- Check '.nextflow.log' file for details

I did try kicking off the pipeline with the current versions of both NF and ampliseq and that seemed to work, but I need the data processed through the same versions as before. Can anyone help me figure out how to solve this? Update: I was able to go into the summary report from my previous run, access the file links used for the DADA2 taxonomy, download those, and point to the local copies for the pipeline. It seems to be working now. I think this should be okay, right? I pulled the references listed in the older run so it should be using the same taxonomy as before along with the same pipeline versions. Does this sound like a good approach?

Some internet servers are inaccessible for certain times, so sometimes databases fail to download. A re-try at another time usually helps in that cases. Manual download also sometimes helps. Additionally, unfortunately links to taxonomic databases are not always stable. We update the links as needed. That means that older pipeline versions cannot always download the requested database. The approach you used is also fine. Just keep in mind that taxonomic classification is not a reproducible process with DADA2, i.e. small differences can occur when float number oscillate around the threshold.

I updated the urls in 2.16.1. For some reason, the urls that work in the browser does not work in Nextflow any longer but the ones in 2.16.1 should work.

Whether this also runs the downstream processes, I don't know.

No it doesnt, because the abundance file is missing. This request to apply downstream analysis to pre-computed ASVs never came up to my knowledge, so not sure that's something expected or wanted.

Of course! Let's wait and see

One more quirk I'm noticing. In the run where I have the DADA2 ref files locally, the pipeline completed and reports having made barplots for all metadata columns, but only one of the output barplot_average folders is actually generated. I've done this twice and it seems to only generate for the last metadata column fed in (or it is deleting/overwriting). Any quick tips on a solution for this?

I don't know much about the downstream processes, but this sounds like a bug (unless @Daniel Straub can explain this better). If you don't get this explained by him, make a GitHub issue where you explain how to reproduce what you're seeing as carefully as possible, e.g. uploading a metadata file that exposes this.

Sure thing! I'll be on the lookout for an explanation and can share my metadata sheet, args, etc if need be 🙂

but only one of the output barplot_average folders is actually generated. I've done this twice and it seems to only generate for the last metadata column fed in (or it is deleting/overwriting).

Hm that might be a bug indeed. We have a CI tests that covers it, but maybe you have some sort of edge case. I had a quick look at the code and I couldnt see any obvious issue.

Any quick tips on a solution for this?

Maybe run once with -

-metadata_category_barplot category1 --outdir results_category1 -resume

followed by

--metadata_category_barplot category2 --outdir results_category2 -resume

so that those barplots are produced in each results folder. The report would not show that however.