Can anyone remind me why we didn't just cram all 4...
# wg-container-conversion-task-force
p
Can anyone remind me why we didn't just cram all 4+ container URIs into some squiggly brackets in the
process.container
directive? Instead of all the stuff with
meta.yml
and creating config files and stuff. I'm pretty sure we discussed it and I almost certainly advocated against it, I'm just struggling to remember why 🙈 😆
m
I think it makes the main.nf clearer
We already have a separate file for the environment.yml, I don't mind a separate file for the containers URI if it makes the main.nf cleaner
p
There is that, but it makes other stuff a lot less clear (multiple pipeline config files with container URIs, modules need meta.yml to be complete, a bunch of extra complexity in the automation)
m
For me, it's more whatever makes it easy for the end user first. So for me as a end user, I honestly don't care much about it. Then what is easier for the dev, and on that side, I think you already answered
p
For the pipeline / module dev it shouldn't matter much as everything should be automated pretty much
It's more @Julian Flesch and @mashehu I worry about 😆
m
Yamls are a lot easier to parse than nextflow might also be a reason
tru 2
p
For some context: I have a long thread with @pontus and was thinking that a dynamic container string could pick the correct container URI at run time based on
task.arch
and whether singularity / apptainer / docker is active.
It'd be an ugly block of code, but it might mean that we didn't need all the pipeline config stuff? 🤔
If it is indeed better then I'm 2 years late with this suggestion lolsob
But like I say, I feel like we discounted it for some reason that I'm forgetting? Need an ADR from 2 years ago!
j
I agree with what was already mentioned and can maybe add to it: • It makes the main.nf more readable • It is easy to parse = lint • Better linting + fixing = More frequent container updates? • It adapts easier to other architectures and engines we might want to add
But I was also a bit late to the container party kickoff. Some modules are not (yet) be suitable for seqera containers. And some stuff we could solve with multi-architecture builds etc.
m
Isn’t there a way of defining
task.ext
attributes from
<http://main.nf|main.nf>
directly ? To separate out defining the actual container paths from the logic of choosing the correct one for the OS / architecture. In Homebrew recipes, there’s a very clear section with all the package builds github.com/homebrew/homebrew-core/blob/…/lizard-analyzer.rb#… In Nextflow modules, that would be followed by a standard
container
line that has all the conditions etc, but only refers to the above variables. It’d be even better if Nextflow had 1st-class support for declining containers …
p
Yeah I have toyed with the idea of extending Nextflow to take either a string or a map for
container
. But new syntax immediately makes adoption tricky 🫠
p
Not sure I follow some of the reasoning above - if we have automation for this, linting essentially goes away?
I do agree on readability, but maybe that can be solved by squinting? 🙂
😂 2
This discussion springs from me wanting to deal with a situation that comes from difficulties with the choice and mixed environment. I'm not sure I see any problem for the "cram it in the container directive" with adaptation to new engines/architectures - that should be straightforward, although not pretty.
For clarity; I'm not saying it's a bad design or needs to be changed, I've mostly been pondering what can be done with where we're at.
Since we're depending on seqera containers, it also depends on where they go I guess - in principal a single repository/tag combination could be used for all architectures for a given engine category (e.g. one string for docker/podman/et.c. and one for singularity/apptainer).
(Not completely sure if all the other bits needed are there, if Seqera containers would get provide that.)
m
Another point: we don't want people to manually edit them, so appending them to a meta.yml (we might even add a disclaimer comment in the future before the section), made more sense than in the middle of the
<http://main.nf|main.nf>