I would like to share <DSO>, a command line helper...
# shameless-promo
g
I would like to share DSO, a command line helper to build fully reproducible data analysis projects with ease. It is very well suited for all custom analyses that come downstream of nf-core pipelines. DSO works by bringing together the best tools for data science: • git, for code versioning • dvc, for data versioning and tracking inputs/outputs for each script • jinja2, for project and stage templates • uv, for Python dependency management • quarto, for authoring reports with R or Python • hiyapyco, for hierarchical YAML configuration files • pre-commit, for consistency checks and linting • automated watermarking of all plots, e.g. to mark reports as preliminary We developed this at Boehringer Ingelheim to meet the high quality standards required for biomarker analysis in clinical trials. We've been using it productively on several projects by now and believe it's useful for a wide range of data analysis projects. If you want to learn more, check out the repo ⭐ and the docs and come attend our bytesize talk nf core party on February 25. DSO was developed as a joint effort with @Daniel Schreyer @Tom Schwarzl @Alex Peltzer
👀 10
👍 3
headbangingparrot 13
p
Nice! Congrats on getting it out 🙂
a
I suspect you may already cover some or all of these in your bytesize. • at the most basic level this is a
dvc
convenience wrapper? • from the toy example, why would one not just roll downstream porcesses e.g
quarto
report generation into another nextflow pipeline? • what are the specific reasons to maintain a parallel R package? • why is
Watermarker
generic
class Watermarker(Generic[ImgType]):
yet never actually seems to be used with
Watermarker[SomeImageType]
as far as I can see in the source code? I think perhaps I am not clear on the optimum/intended use-case here. As someone who has primarily developed pipelines to process and return clinical data I don’t understand why I would want to version inputs/outputs alongside my project workflow code. I wonder if this sort of thing might be useful in conjunction with Maxime’s
nf-core/references
(i.e. using
dvc
as a data registry and consistently updating outputs as changes are made to the core pipeline)
g
I indeed plan to talk about most of the points in the talk, but I'll after briefly here:
at the most basic level this is a dvc convenience wrapper?
yes. to draw a parallel, i'd say dso relates to dvc as nf-core tools relates to nextflow
from the toy example, why would one not just roll downstream porcesses e.g quarto report generation into another nextflow pipeline?
I did something like that in the past (e.g. https://github.com/icbi-lab/luca). It worked well, but it's a lot more overhead to write a nextflow pipeline than writing dvc stages. You also get more features which is nice if you have a really data intensive workflow, but most of the time it's not needed. So I think dvc has a lower barrier to use for data analysis than nextflow. DVC also provided data versioning and provenance (it tracks the version of input files, scripts and generated output files in lock files), which is not that straightforward to achieve with nextflow. Some also say downstream stuff should be in a generic nextflow pipeline (such as #C045UNCS5R9). These workflows are great for some type of standardized analyses, but IMO don't offer the flexibility needed in many cases.
what are the specific reasons to maintain a parallel R package?
The R package is a minimal helper package just for convenience. The main function is
read_params
which allows to read the compiled parameters from a stage and always gets the path right independent of your working directory. If you want, you can do without.
why is Watermarker generic class Watermarker(Generic[ImgType]): yet never actually seems to be used with Watermarker[SomeImageType] as far as I can see in the source code?
Good catch, that seems to be a leftover of an earlier implementation.
I think perhaps I am not clear on the optimum/intended use-case here. As someone who has primarily developed pipelines to process and return clinical data I don’t understand why I would want to version inputs/outputs alongside my project workflow code.
It's not for the "standard" usecases. Nextflow pipelines are great for that. It's more for exploratory analyses.
👍 1
a
Thanks for that. I will try to catch the bytesize to get a better grasp too.
m
Sounds super cool! Reminds me a bit of https://cookiecutter-data-science.drivendata.org/
👍 1