Got a suggestion, due to colleagues returning to pipelines with modules that download dbs from urls after a long time, and resume is broken. How do you all feel about a predownload workflow under workflows/ ? That would be execute-once (for user-selected dbs), and then the main workflow would only allow the _path versions of those parameters.

I feel that any pipeline that downloads reference files should have an option for you to store them

how did they get by the offline CI?

I guess you could set up some way to pass your already downloaded file through the pre-workflow to give it the path, after validating it exists?

although if you are expecting

-resume

to work after more than a couple of weeks you are probably optimistic 😉

We have established patterns for this for reference genomes already, I guess this should be the same?

If db input can be specified via the input of a DB-specific sample sheet, that might be a nice way to handle the pipeline downloads

It was funcscan @Rike Hanssen..what is an offline CI? :p I've never used pipelines that use genomes, so I am not sure how this is resolved for that case.

I'm being parent taxi ATM but I am confused about this thread, will get back a bit later

There is a CI that hsould test offline mode: I guess it jsut tests the download: github.com/nf-core/tools/blob/…/download_pipeline.yml But in principle all pipelines should be capable of running offline: nf-co.re/usage/offline so it should be possible to disable outreach to external servers by supplying all references etc. Otherwise people in air-gapped environments can't run the pipeline

sigh kids.... But nice for me, @Rike Hanssen basically say what I was going to say. We generally have always had the pattern anyway that when you have a reference or database being used, if the pipeline developer adds a download module then there should be a parameter that allows you to supply a pre downloaded copy for future runs. Then you should make it very clear in the documentation that you should move the pipeline downloaded one to a safe place for reuse (and document where to get the databases outside the pipeline) This has been around since eager days.... It's nothing new. The offline mode indeed just tests that

nf-core download

works, and I imagine most pipelines can run without a reference database (or maybe the CI also downloads the test data locally itself?

I get the point, but it is a bit sad that if the first time you used the url and downloaded the db, and then in the future you decide to switch to the downloaded path (because url cache fails or whatever), the resume fails and reruns everything. I haven't tested the offline mode though, but if that works, the fix is already there. In my personal pipelines, I think I will separate the download logic though to its own workflow. I agree with James's points though for nf-core ones

But wouldn't it fail to resume anyway regardless?

If the db was predownloaded, and then the _path version of the parameter was provided in the first main workflow run, next time (if all inputs have remained in place ofc) cache wouldn't break, no.

Ah ok, so you are only referring to the breaking of the resume of the very first run then?

Yep, exactly. For example somebody runs the pipeline for the first time using the url params, and completes it to the very end, for 100 samples. Then comes back in the future to test/benchmark different params for a module in the middle of the pipeline. Heartbreak

I think @Daniel Lundin is using

storeDir

for this. I think it might be the only pipeline that is utilizing this a lot, so there might be gremlins somewhere but it looks like it might solve this problem to some extend.

I haven't read the full thread, but

storeDir

works like a charm in both magmap and metatdenovo. The only problem is versions since the

eval

style doesn't work and to allow users to download data outside the pipeline requiring a

versions.yml

is awkward. It was @Mahesh Binzer-Panchal who gave us the tip originally.

Yes,

storeDir

was the way to do this: https://nfcore.slack.com/archives/CE6SDBX2A/p1772801992798349?thread_ts=1772799969.096169&cid=CE6SDBX2A If the files are already there, the process doesn't activate. It treats it as a form of

cache

. ( first time as well as on

-resume

). But then it's not compatible with

eval

. So now you have to supply a parameter with the database path, and basically programmatically do what

storeDir

did. See the GH thread on this with Ben github.com/nextflow-io/nextflow/issues/5785#…

workflow {
    index_path = file("${params.store_dir}/genome.bwt")

    index = index_path.exists()
        ? Channel.fromPath("${params.store_dir}/genome.*").collect()
        : BUILD_INDEX().out.index

    ALIGN(reads, index)
}

Yep, exactly. For example somebody runs the pipeline for the first time using the url params, and completes it to the very end, for 100 samples. Then comes back in the future to test/benchmark different params for a module in the middle of the pipeline.

Meh I think for a SINGLE run to hurt if they didn't read the docs is not worth it... it's a small amount of pain IMO rather than learning a whole other pipeline for a one off thing

As long as you used a parameter to point to the db location, you could always have it in a shared location.

storeDir

would check that path. It never needed to be run first

Ok maybe that was the issue when I reviewed. That sounds like a good trick though, I wonder if this something we could add in some specificationss somewhere?

I actually never understood your concerns @James Fellows Yates. We use this in two slightly different ways. The first is for databases that you download once and potentially reuse. Nothing strange, and it fits any use where you want to share the database between different projects. Handy at an HPC where storage is often limiting. The other is in magmap, where the pipeline with time builds a "database" of downloaded genomes and annotations. In one machine I now have 80,000 genomes and ecah time I run the pipeline it's likely to want to reuse some of them, but not all, so the databases will grow incrementally. No code trickery required, the pipeline just notices: 30,000 of those I already have, let's download the other 5.000. And, another advantage, no process call will be submitted to the queue.

@Daniel Lundin ok the maybe I misunderstood at the time! Apologies!

No problem!