Hi everyone, We are almost ready for the first release of #C07LY51P4S2 - our PR has already been approved, but I've just realised we didn't request data upload for the AWS Megatests. But before filling in the request form, I have a couple of questions: • Our pipeline, besides using local files as input, also pulls genomes from ncbi RefSeq/GenBank accessions using the

ncbigenomedownload

module. Would a samplesheet containing accessions suffice for

test_full

? Or do I have to request these genomes to be uploaded and pulled from the S3 bucket? • To fully run this pipeline (optional subworkflows included), we need two databases: the Dfam database for transposable elements detection and the FCS-GX database for contamination screening. The Dfam database would require two H5 partitions, which in total is ~60GB. But the FCS-GX one is 500GB, and I don't think I can make it smaller. Should we only request to upload the partitions from the Dfam database? The FCS-GX subworkflow would remain untested.

1. If you can directly reference genbank etc. URLs and the profile works, please do this. That's the preferred method. Copying input files into the s3 bucket is a last resort to solve flaky downloads and annoying failures (once using s3 it costs us AWS credits for data egress to use - not loads, but the less the better). 2. As above: can folks get these from some other source?

1. Great, I'll just have a samplesheet with the RefSeq accessions then. 2. We have modules to download both databases , so we can simply point to the links in the

test_full.config

. Not sure if it's wise to do it for the FCS-GX full database due to it's size? There's also a FCS-GX test database which is only ~1GB that we can use instead. The contamination screening subowrkflow will run, but of course it won't show any results as this test was made for a specific assembly

I don't really follow on (2) sorry. Module downloads it from where? But yeah downloading .5 TB per run will cost quite a lot in data egress so I'm not super keen.

Sorry, I didn't explain myself very well. We have two modules that can fetch databases from download links: • For Dfam subworkflow: Fetches the Dfam H5 partitions from the Dfam server (dfam.org/releases/Dfam_3.9/families/FamDB). For

test_full

input assemblies, we'd need partitions

0

and

3

(~80 GB total) for the TE detection subworkflow. I could point to these in the

test_full.config

and the pipeline will download them, instead of having them in the bucket. • For FCS-GX subworkflow: This is an official nf-core module. Fetches the NCBI GX database from the NCBI server (ftp.ncbi.nlm.nih.gov/genomes/…/latest). The full DB is ~470 GB. There's also a ~1 GB test-only DB, which allows the subworkflow to run end-to-end but doesn't produce a meaningful result. I think running this on the full database might be too much, so I'd rather just use the ~1 GB one. These would be the alternatives to having these databases in a bucket.

Great! Then yes, I think start off with those if that's ok.