Yesterday in the <#C070369GP7T|meta-omics> meeting...
# tools
j
Yesterday in the #meta-omics meeting it was brought up by @Samuel Lampa that it would be nice to have better support for offline pipelines when a pipeline has auxiliary/external databases it needs to download (e.g. for particular tools) - along the lines of a
requirements.txt
but for external data that gets executed e.g. during
nf-core pipelines donwload
. Someone mentioned (I think it was @Daniel Lundin) that at some point there was a discussion of something along those lines... did that go anywhere? I think it's a cool idea
πŸ‘ 1
m
I am not aware of this discussion and also not with regard to
nf-core pipelines download
. I think, one fundamental difference of the #meta-omics perspective vs. that of many other fields is, that they tend to work with large reference databases that that collect signatures of taxons, so that reference will be more or less identical for most use cases. One tool has basically one pre-built default reference (plus maybe a smaller and a more comprehensive version of it as well). In contrast elsewhere, you usually deal with highly heterogeneous references ranging from prokaryotic to diverse eukaryotic species, such that just downloading the default that the pipeline maintainer has chosen won't help you anyway. Maybe in the future the #references workflow can simplify some of that, but ask Maxime how complex that problem when going after the nitty-gritty details. In the past,
nf-core pipelines download
has already struggled to keep up with new templates, new ways to declare containers etc. The heterogeneity of parsing the
test_full
profiles and trying to automatically derive which one of those is a relevant reference to download is staggering. Instead, I suggest pipeline authors who wish to support something like that to take a look at #sarek . This workflow has nice support to run it just for creating a local cache of the VEP and SNPeff references.
j
OK, then maybe I was mistaken/misinterpreted re the
download
subcommand. Actually in this case #references wouldn't help, as this is still a workflow that would have to be executed with expectation it is running like any other nextflow pipeline (submitted to a cluster). You could e.g. run it on a head node with local mode but it's much harder to control interms of resources etc. I think what we were discussing/thinking is more along the lines of a conda
build.sh
type thing which has much less computational overhead, where you could execute a bunch of 'additional' commands when downloading. Something like when you run
nf-core pipelines download
, after cloning the repository it looks for a script file like
fetch-data.sh
which the pipeline developer has defined to pick up all relevant databases etc. But yes, I do see that maybe this type of thing may only be useful for a small set of pipelines.
(if you follow... ) but regardless if there wasn't a previous discussion maybe it's a moot point. @Samuel Lampa could still make an issue to
tools
though πŸ™‚
m
> Something like when you run
nf-core pipelines download
, after cloning the repository it looks for a script file like
fetch-data.sh
which the pipeline developer has defined to pick up all relevant databases etc. But yes, I do see that maybe this type of thing may only be useful for a small set of pipelines. If it anyway is a bash script, then one can pretty easily execute it outside of
nf-core download
as well. And if it is a long running command because it is downloading hundreds of Gigabytes, you will anyway want to run it in a
screen
session or submit it to Slurm. Download has support for background processes, but it would still block the terminal while running. I was fearing you wanted to add a new value like
cachable
to the
nextflow_schema.json
that would then indicate that this parameter specifies a path that could be downloaded. > Samuel Lampa could still make an issue to
tools
though πŸ™‚ Sure. To increase the chances of implementation, pull requests are usually more appreciated, though πŸ˜‰. The backlog in the tool is huge and we already have a big refactor ongoing that is somewhere 40% through after #hackathon-mar-2025-workflows-to-go, but currently on hiatus.
j
If it anyway is a bash script, then one can pretty easily execute it outside of nf-core download as well. And if it is a long running command because it is downloading hundreds of Gigabytes, you will anyway want to run it in a screen session or submit it to Slurm. Download has support for background processes, but it would still block the terminal while running.
True, but more work for a user which I guess is the issue (some pipelines do describe these steps in the docs, but still a bit tedious).
I was fearing you wanted to add a new value like cachable to the nextflow_schema.json
that would then indicate that this parameter specifies a path that could be downloaded. Nah, the other reason this doesn't fully work as some tools force you do use their own internal download tool that does extra processing twitching eye, so a simple download isn't enough. > Sure. To increase the chances of implementation, pull requests are usually more appreciated, though πŸ˜‰. The backlog in the tool is huge and we already have a big refactor ongoing that is somewhere 40% through after #hackathon-mar-2025-workflows-to-go, but currently on hiatus. (edited) Well of course, but I'm pretty sure that would've been done already if it was possible by the person who was proposing it 😬 πŸ˜‰
m
> True, but more work for a user which I guess is the issue (some pipelines do describe these steps in the docs, but still a bit tedious). Sorry, but it is not my job to spent numerous hours developing a tool, so that users can skip a few minutes reading the manual that some other poor person took much longer to write as well. tux sorry (_I hope that doesn't come over too rude. I am all for great UX, but bioinformatics is simply not like an ordinary mobile app with a limited number of functionalities that can be accessed from a few buttons and swipes. Some form of knowledge and effort is unfortunately still required. nf-core pipelines are not magic bioinfo things that output the analyzed data for you with the press of one button, but tools for experts who know what they are doing and wish to scale their analyses. For the 'press one button' folks you have Seqera Studio, Plutio.bi_o and Mithrl and so on but then they have to pay for the expertise in the background and the convenience.) > Nah, the other reason this doesn't fully work as some tools force you do use their own internal download tool that does extra processing twitching eye, so a simple download isn't enough. Which would then clearly argue for the #sarek solution, because if #tools would need to launch the tool itself, e.g. within a container, then it should directly be done within Nextflow. ‼️
s
Hi, and thanks @James Fellows Yates for bringing this up, and @Matthias Zepper for the responses! I can perhaps add that while not sure on the optimal solution, I see two problems with various implicit downloads happening in different places of the pipeline: 1. It is easy to miss some external dependency which will make a pipeline not work in offline mode, just because you almost always have internet access while testing it (there might be a download in a branch of the workflow logic that is seldom executed). 2. For upcoming regulations in clinical settings like IVDR, it is central to be able to provide full provenance of all the software and data used. Here, it would help tremendously to have some kind of machine-parsable list of the external dependencies. All in all, I think the angle I'm coming from is more about explicit declaration of online data dependencies, than the tooling itself, although appropriate tooling might sure help too, and would perhaps motivate using (and continuously) the declaration info too.
j
Sorry - yes I went a bit further than what you yourself proposed based on what Daniel had said (But he openly said he wasn't πŸ’― sure)
s
No worries! I'm happy you brought it up, and only natural to develop it further πŸ™‚
πŸ‘ 1
j
Mainly I went further as:
All in all, I think the angle I'm coming from is more about explicit declaration of online data dependencies, than the tooling itself, although appropriate tooling might sure help too, and would perhaps motivate using (and continuously) the declaration info too.
An explicit online data dependency if you require a specific tool's subcommand to download the data may not work... unless you go something like a JSON thing where you have attributes for 'location of file' and 'command used to download' and 'version of tool'
Copy code
[antismash]
name = 'antismash_v7'
version = 7
location = ftp://blah.com/database/antismash_v7.tar.gz ## Assuming there is a singular location, which is the main issue I think
tool = 'antismash download_databases'
tool_version = '7.1.0'
Is that sort of what you meant @Samuel Lampa?
m
I have not used it so far, but isn't that what the RO Crate command in tools does for the workflow itself and the nf-prov(enance) plugin for a particular run of a workflow?
j
Yes, similar concept
s
@James Fellows Yates Yeah, this is an issue, with tool-specific download commands, and I don't know what would be the best solution here. I'll need to think a bit about that one! @Matthias Zepper, yes, we concluded yesterday also that I should look closer into RO Crate before going too much further (have looked briefly before, but need to take another look)
πŸ‘ 1
j
I guess this would require sort of like an ROcrate like thing within an ROcrate, but can be parsed and executed by something
(to go the full solution as I was pondeirng, not just provnenance tracking)
s
Yes. And to add on to the RO Crate thing, I also want to stress that I think it would be preferable to go towards a situation where provenance and definition of what to execute would be marred closer to each other. That would ensure provenance is never an afterthought, but built in to the way things work.
πŸ‘ 1
m
Maybe you guys could work this out and either make a Bytesize talk out of it or take it to #regulatory? I didn't mean to be too dismissive, just wanted to say that we at NGI have probably no use for it, so won't invest much time developing it. But generally any improvements to nf-core are of course welcome!
πŸ‘ 2
s
(As a case in point, for my experimental scicommander provenance tracking tool, I used TES format (Task Execution Schemas, which are for executions) also for storing provenance. Meaning you are ready to re-execute this task already, since you have the TES ready, with just some additional annotations for timing, dates etc)
Wanted to get much more involved with RO Crate and nf-prov already before ... but been so hard to find the time. But maybe it is time now ...
πŸŽ‰ 1
🀘 1
j
I am just an observer, so I would leave @Samuel Lampa to decide that - I wouldn't have use for it myself. But to be clear: I never meant that I was expected/assuming this would your task @Matthias Zepper nor NGIs! I'm sure other people would need
download
for offline contexts. Apologies if you thought that was implied when I mentioned
download
-> I know you're the main architect of that but anything tools I normally just throw it into the ether to see who bites πŸ˜…
m
> but anything tools I normally just throw it into the ether to see who bites At least for the last two years,
download
tasks didn't match the bait taste of anyone else but me, and that was more of escalating commitment hostage situation than anything else. 😬 During the last Hackathon, I at least lured some people to work on the code, so maybe you are indeed no longer fishing in a gummy pond with that. But there is an issue backlog of 3 years at the moment as well...so maybe it just adds to an every growing pile.
πŸ‘ 1
m
We’ve had a similar internally discussion for our pipelines. Several need databases to be downloaded, sometimes pre-processed in a certain way. We’ve considered several options using Nextflow for that (because some pre-processing steps are expensive and should be offloaded to the HPC) but haven’t settled on / implemented any yet: 1. extra
workflow
entry in the pipeline named
SETUP
or something 2. the default workflow transparently downloads everything if the input database parameters are empty 3. separate setup pipeline that downloads data and outputs a config file with the locations that can be passed to the main pipeline with
-c
It can easily take hours to download all the dbs so we’re wary of solution 2 as users who don’t pay attention may end up making 100s of GBs again and again. Solution 3 has got the advantage we could have a single setup pipeline for all our analysis pipelines.
πŸ€” 1
p
I think that we discussed having a general solution for this as part of the #references work (maybe @Maxime Garcia knows?)
m
For me that's outside the current scope of references, but we can always evolve
j
I agree, not really in scope
There is no building per se, simply downloading
m
Let's brainstorm next week and see what we can think of
p
But references will consist of a big YAML file with a spec saying where loads of files are, right? So shouldn't be too difficult to build some kind of tooling off that for downloads, I was thinking
j
That works for so singular files, but what if a tool has a dedicated download command thingy that unpacks stuff in a special way or w/e
m
> That works for so singular files, but what if a tool has a dedicated download command thingy that unpacks stuff in a special way or w/e How about a dedicated module or subworkflow per tool? Then it could... β€’ Be used in a pipeline-specific reference download, like Sarek does with the SNPEff and VEP references. β€’ Reused in other nf-core pipelines, potentially with reference-specific customization. β€’ Integrated into other, potentially institution specific reference-gathering workflows maintained outside of nf-core, but using the nf-core tooling. The advantage of that approach would be, that we would not need to devise a one size fits all solution, but could add support for reference gathering tool by tool like we gradually port pytest to nf-test module by module. The only argument I see for having it in #tools is if there were certain things that are easier implemented in Python than in Nextflow.
m
The only argument I see for having it in #tools is if there were certain things that are easier implemented in Python than in Nextflow.
Then Nextflow it is πŸ˜†
sarek cat jam 1
j
I think having
download
to run anextflow pipeline while it's downloading is a) overkill, and b) not actually suitable for the reason I said above - I don't think you can restrict the number of CPUs a whole run can use which would be very bad when running on the internet connected 'head node' . Thing is most pipelines do already auto-download databases if you don't supply it - and this is a very nice experience for people with internet connction, moving this to a whole separate workflow would be not a nice dev experience. BUT DESPITE ALL OF THIS I'm digging myself into a deep hole of something I myself have 0 interest inusing πŸ˜†, soi I'm going to let this thread lie now 😬
m
I think having
download
to run anextflow pipeline while it's downloading is a) overkill, and b) not actually suitable for the reason I said above
I have never suggested that. #tools or download will not be involved.
Thing is most pipelines do already auto-download databases if you don't supply it - and this is a very nice experience for people with internet connection, moving this to a whole separate workflow would be not a nice dev experience.
But is it really so hard to have a run mode of the pipeline where all those auto-download steps run at the very beginning of a pipeline? So people can run it elsewhere to auto-download the databases to a parameter-specified location?