Gregor Sturm
02/04/2025, 7:43 AMPhil Ewels
02/04/2025, 8:55 AMArthur
02/04/2025, 9:04 AMdvc convenience wrapper?
• from the toy example, why would one not just roll downstream porcesses e.g quarto report generation into another nextflow pipeline?
• what are the specific reasons to maintain a parallel R package?
• why is Watermarker generic class Watermarker(Generic[ImgType]): yet never actually seems to be used with Watermarker[SomeImageType] as far as I can see in the source code?
I think perhaps I am not clear on the optimum/intended use-case here. As someone who has primarily developed pipelines to process and return clinical data I don’t understand why I would want to version inputs/outputs alongside my project workflow code.
I wonder if this sort of thing might be useful in conjunction with Maxime’s nf-core/references (i.e. using dvc as a data registry and consistently updating outputs as changes are made to the core pipeline)Gregor Sturm
02/04/2025, 10:10 AMat the most basic level this is a dvc convenience wrapper?yes. to draw a parallel, i'd say dso relates to dvc as nf-core tools relates to nextflow
from the toy example, why would one not just roll downstream porcesses e.g quarto report generation into another nextflow pipeline?I did something like that in the past (e.g. https://github.com/icbi-lab/luca). It worked well, but it's a lot more overhead to write a nextflow pipeline than writing dvc stages. You also get more features which is nice if you have a really data intensive workflow, but most of the time it's not needed. So I think dvc has a lower barrier to use for data analysis than nextflow. DVC also provided data versioning and provenance (it tracks the version of input files, scripts and generated output files in lock files), which is not that straightforward to achieve with nextflow. Some also say downstream stuff should be in a generic nextflow pipeline (such as #C045UNCS5R9). These workflows are great for some type of standardized analyses, but IMO don't offer the flexibility needed in many cases.
what are the specific reasons to maintain a parallel R package?The R package is a minimal helper package just for convenience. The main function is
read_params which allows to read the compiled parameters from a stage and always gets the path right independent of your working directory. If you want, you can do without.
why is Watermarker generic class Watermarker(Generic[ImgType]): yet never actually seems to be used with Watermarker[SomeImageType] as far as I can see in the source code?Good catch, that seems to be a leftover of an earlier implementation.
I think perhaps I am not clear on the optimum/intended use-case here. As someone who has primarily developed pipelines to process and return clinical data I don’t understand why I would want to version inputs/outputs alongside my project workflow code.It's not for the "standard" usecases. Nextflow pipelines are great for that. It's more for exploratory analyses.
Arthur
02/04/2025, 10:13 AMmoritz
02/04/2025, 10:58 AM