@Rike Hanssen: I've been following the AWS run you initiated. It fails on creating the directories for contigs and prokka output respectively:

Unexpected path for Fusion script launcher: /magmap_genomes

. I suppose we need to set the params for these two differently in the full test, can you advice us? Basically, it's just two directories that start empty and will be populated by the run. If they don't exist when the pipeline starts, the pipeline attempts to create them. I completely inexperienced when it comes to AWS directories and cloud computing in general.

i think the

genome_store_dir

will need to be an s3 path. maybe easiest, if you just prepend it by

params.outdir

then it should always be fine to be created regardless of what systems users have.

Sorry for the late reply; I was in meetings. I'm not sure one can use the same directory. The reason is the

versions.yml

that is also copied to the store directory. This at least messes things up with nf-tests. But, never mind, the important thing is that we set the

--genome_store_dir

and

--prokka_store_dir

params in the full test to paths that work. And, just to make sure we don't run into trouble with the

versions.yml

I'd prefer it to be different directories. If it somehow makes things easier, they might well be subdirectories of the output directory. What can I set them to?

https://cloud.seqera.io/orgs/nf-core/workspaces/AWSmegatests/watch/PNQVk85Pg44wc/v2/configuration here i did `params.outdir`/prokka_genomes etc. although this run is now failing on something

Right. I also just saw that it fails. I've never seen it fail before in the

PROKKA

process.

minus()

is used to remove part of the file name and it hence seems like the file name is null. Maybe one can just resume? If it happens again, I suppose we need to check that null file names -- possibly resulting from failed fetches of genomes -- are never passed to the process.

sure. I can do that

Thanks

we could upload the data to the s3 bucket and provide it from there, if your full size test can also stage in genomes directly

The first part of that was what I was thinking of, my question being how I can practically do that. The second part, yes, the pipeline actually downloads all genomes and stores them in the store directory so after one run they should be ready for the second. (This also applies to the prokka step.) My updated question then becomes: Is it OK to let this first run do the downloads and genomes will then be kept or would it be better to interrupt, transfer the genomes (including annotations) and then restart?

the way we are storing the genomes at the moment in the outdir location which is unique to each release. we should store these similarly like we store input data for other pipelines. let's see if this run finished now and then I can transfer them directly on AWS, if that doesn't work then we should do this manually

We got the same error as before. This is strange. The pipeline should retry failed downloads five times then ignore, so this shouldn't happen. Maybe it would be better if the genomes were first uploaded. I can provide the data but since it's quite a lot I'm not sure which is the best way.

Does the caching work locally? cause non of the prokka genome downloads were cached here

The caching works great both for genomes and annotations when running the pipeline on HPC or individual machines.