A new request to upload a dataset to nf-core AWS S3 bucket has been made! • Requested by: @Niklas Schandry • Pipeline name: genomeassembler • Why is this needed?: This is HiC test data for genomeassembler • Size of the dataset: 5,749.57 MB for _1 and 6,069.85 MB for _2 • URL: GSA browers: https://ngdc.cncb.ac.cn/gsa/browse/CRA004538/CRR302669 Direct file (https) https://download.cncb.ac.cn/gsa/CRA004538/CRR302669/CRR302669_f1.fastq.gz https://download.cncb.ac.cn/gsa/CRA004538/CRR302669/CRR302669_r2.fastq.gz • Related files: Core team will add this dataset to the nf-core bucket. Please react to the message with ✅ when this is done. [This process is not automated]

Hi @Niklas Schandry - is this needed, or can the tests pull directly from those locations?

yeah, ftp might be better

I will give it a try

I guess in case this comes up again, it seems that ftp for cncb.ac.cn is more reliable

I don't think that can be right. That would mean that basically no nf-core pipelines ever resume

The tasks where the inputs were already on s3 are fine, the task that has a foreign input does not resume

I have a new feature for Platform in development at the moment which would be perfect for this: it takes a task hash components and gives you a diff against any other task (eg. one you expected it to resume from)

To me the giveaway is `Staging foreign file: ftp://download.big.ac.cn/gsa/CRA004538/CRR302669/CRR302669_f1.fastq.gz`in the resumed run log

Sure, but that'll happen if the task is not cached. It doesn't necessarily mean that the task was not cached because of the file staging.

I see, that makes sense. So files that are not inputs are not staged, is there a way around that?

is there a reason not to stage them?

I am not even sure if I would see that in local tests, since in our hpc, all nodes see scratch, where work is, and can probably resolve the path fine

That's generally best practice, to treat the work dir as a little encapsulated bubble. Everything going in and out is then tracked.

Mainly that I kind of like what I am doing now and patching the module is annoying 😉

I suspect that what you're doing now works fine on HPC and local, but will break on any and every cloud / object storage

I need to have a think if declaring those as inputs fundamentally breaks my general approach for that

Without having really looked much, I assume that they are similar to reference genome inputs used in other pipelines?

Yeah should be ok

uff, it's the quast module?

Yup

That module is in several other pipelines, are they not affected in the same way?

Only if they do the same weird thing I did 😄

Fair enough 👍🏻 Ideal situation is to have it in the module as an optional input in this case I think

I think for tools like Quast, which offer a ton of optional inputs (14 optional paths?), providing a minimal set of inputs and patching at the pipeline level as required is more pragmatic.

Fair enough 👍🏻