James Fellows Yates
04/11/2025, 6:21 AMrequirements.txt but for external data that gets executed e.g. during nf-core pipelines donwload. Someone mentioned (I think it was @Daniel Lundin) that at some point there was a discussion of something along those lines... did that go anywhere? I think it's a cool ideaMatthias Zepper
04/11/2025, 6:32 AMnf-core pipelines download.
I think, one fundamental difference of the #meta-omics perspective vs. that of many other fields is, that they tend to work with large reference databases that that collect signatures of taxons, so that reference will be more or less identical for most use cases. One tool has basically one pre-built default reference (plus maybe a smaller and a more comprehensive version of it as well).
In contrast elsewhere, you usually deal with highly heterogeneous references ranging from prokaryotic to diverse eukaryotic species, such that just downloading the default that the pipeline maintainer has chosen won't help you anyway. Maybe in the future the #references workflow can simplify some of that, but ask Maxime how complex that problem when going after the nitty-gritty details.
In the past, nf-core pipelines download has already struggled to keep up with new templates, new ways to declare containers etc. The heterogeneity of parsing the test_full profiles and trying to automatically derive which one of those is a relevant reference to download is staggering.
Instead, I suggest pipeline authors who wish to support something like that to take a look at #sarek . This workflow has nice support to run it just for creating a local cache of the VEP and SNPeff references.James Fellows Yates
04/11/2025, 6:37 AMdownload subcommand.
Actually in this case #references wouldn't help, as this is still a workflow that would have to be executed with expectation it is running like any other nextflow pipeline (submitted to a cluster). You could e.g. run it on a head node with local mode but it's much harder to control interms of resources etc.
I think what we were discussing/thinking is more along the lines of a conda build.sh type thing which has much less computational overhead, where you could execute a bunch of 'additional' commands when downloading.
Something like when you run nf-core pipelines download, after cloning the repository it looks for a script file like fetch-data.sh which the pipeline developer has defined to pick up all relevant databases etc. But yes, I do see that maybe this type of thing may only be useful for a small set of pipelines.James Fellows Yates
04/11/2025, 6:38 AMtools though πMatthias Zepper
04/11/2025, 6:47 AMnf-core pipelines download, after cloning the repository it looks for a script file like fetch-data.sh which the pipeline developer has defined to pick up all relevant databases etc. But yes, I do see that maybe this type of thing may only be useful for a small set of pipelines.
If it anyway is a bash script, then one can pretty easily execute it outside of nf-core download as well. And if it is a long running command because it is downloading hundreds of Gigabytes, you will anyway want to run it in a screen session or submit it to Slurm. Download has support for background processes, but it would still block the terminal while running.
I was fearing you wanted to add a new value like cachable to the nextflow_schema.json
that would then indicate that this parameter specifies a path that could be downloaded.
> Samuel Lampa could still make an issue to tools though π
Sure. To increase the chances of implementation, pull requests are usually more appreciated, though π. The backlog in the tool is huge and we already have a big refactor ongoing that is somewhere 40% through after #hackathon-mar-2025-workflows-to-go, but currently on hiatus.James Fellows Yates
04/11/2025, 6:55 AMIf it anyway is a bash script, then one can pretty easily execute it outside of nf-core download as well. And if it is a long running command because it is downloading hundreds of Gigabytes, you will anyway want to run it in a screen session or submit it to Slurm. Download has support for background processes, but it would still block the terminal while running.True, but more work for a user which I guess is the issue (some pipelines do describe these steps in the docs, but still a bit tedious).
I was fearing you wanted to add a new value like cachable to the nextflow_schema.jsonthat would then indicate that this parameter specifies a path that could be downloaded. Nah, the other reason this doesn't fully work as some tools force you do use their own internal download tool that does extra processing twitching eye, so a simple download isn't enough. > Sure. To increase the chances of implementation, pull requests are usually more appreciated, though π. The backlog in the tool is huge and we already have a big refactor ongoing that is somewhere 40% through after #hackathon-mar-2025-workflows-to-go, but currently on hiatus. (edited) Well of course, but I'm pretty sure that would've been done already if it was possible by the person who was proposing it π¬ π
Matthias Zepper
04/11/2025, 7:02 AMSamuel Lampa
04/11/2025, 7:21 AMJames Fellows Yates
04/11/2025, 7:22 AMSamuel Lampa
04/11/2025, 7:23 AMJames Fellows Yates
04/11/2025, 7:32 AMAll in all, I think the angle I'm coming from is more about explicit declaration of online data dependencies, than the tooling itself, although appropriate tooling might sure help too, and would perhaps motivate using (and continuously) the declaration info too.An explicit online data dependency if you require a specific tool's subcommand to download the data may not work... unless you go something like a JSON thing where you have attributes for 'location of file' and 'command used to download' and 'version of tool'
[antismash]
name = 'antismash_v7'
version = 7
location = ftp://blah.com/database/antismash_v7.tar.gz ## Assuming there is a singular location, which is the main issue I think
tool = 'antismash download_databases'
tool_version = '7.1.0'
Is that sort of what you meant @Samuel Lampa?James Fellows Yates
04/11/2025, 7:35 AMSamuel Lampa
04/11/2025, 7:36 AMJames Fellows Yates
04/11/2025, 7:36 AMJames Fellows Yates
04/11/2025, 7:37 AMSamuel Lampa
04/11/2025, 7:38 AMMatthias Zepper
04/11/2025, 7:40 AMSamuel Lampa
04/11/2025, 7:41 AMSamuel Lampa
04/11/2025, 7:43 AMJames Fellows Yates
04/11/2025, 7:43 AMdownload for offline contexts. Apologies if you thought that was implied when I mentioned download -> I know you're the main architect of that but anything tools I normally just throw it into the ether to see who bites π
Matthias Zepper
04/11/2025, 7:55 AMdownload tasks didn't match the bait taste of anyone else but me, and that was more of escalating commitment hostage situation than anything else. π¬
During the last Hackathon, I at least lured some people to work on the code, so maybe you are indeed no longer fishing in a gummy pond with that. But there is an issue backlog of 3 years at the moment as well...so maybe it just adds to an every growing pile.Matthieu Muffato
04/11/2025, 9:09 AMworkflow entry in the pipeline named SETUP or something
2. the default workflow transparently downloads everything if the input database parameters are empty
3. separate setup pipeline that downloads data and outputs a config file with the locations that can be passed to the main pipeline with -c
It can easily take hours to download all the dbs so weβre wary of solution 2 as users who donβt pay attention may end up making 100s of GBs again and again.
Solution 3 has got the advantage we could have a single setup pipeline for all our analysis pipelines.Phil Ewels
04/12/2025, 11:09 PMMaxime Garcia
04/13/2025, 7:17 AMJames Fellows Yates
04/13/2025, 7:18 AMJames Fellows Yates
04/13/2025, 7:18 AMMaxime Garcia
04/13/2025, 7:19 AMPhil Ewels
04/13/2025, 7:32 AMJames Fellows Yates
04/13/2025, 9:10 AMMatthias Zepper
04/13/2025, 1:23 PMMaxime Garcia
04/13/2025, 3:17 PMThe only argument I see for having it in #tools is if there were certain things that are easier implemented in Python than in Nextflow.
Then Nextflow it is π
James Fellows Yates
04/14/2025, 7:14 AMdownload to run anextflow pipeline while it's downloading is a) overkill, and b) not actually suitable for the reason I said above - I don't think you can restrict the number of CPUs a whole run can use which would be very bad when running on the internet connected 'head node' .
Thing is most pipelines do already auto-download databases if you don't supply it - and this is a very nice experience for people with internet connction, moving this to a whole separate workflow would be not a nice dev experience.
BUT DESPITE ALL OF THIS I'm digging myself into a deep hole of something I myself have 0 interest inusing π, soi I'm going to let this thread lie now π¬Matthias Zepper
04/14/2025, 8:06 AMI think havingI have never suggested that. #tools or download will not be involved.to run anextflow pipeline while it's downloading is a) overkill, and b) not actually suitable for the reason I said abovedownload
Thing is most pipelines do already auto-download databases if you don't supply it - and this is a very nice experience for people with internet connection, moving this to a whole separate workflow would be not a nice dev experience.But is it really so hard to have a run mode of the pipeline where all those auto-download steps run at the very beginning of a pipeline? So people can run it elsewhere to auto-download the databases to a parameter-specified location?