Hi! I was wondering if it was possible to mirror these docker containers / hosts them on quay.io?

<http://docker.io/quentinblampey/sopa:2.1.11|docker.io/quentinblampey/sopa:2.1.11>

,

<http://docker.io/quentinblampey/sopa:2.1.11-baysor|docker.io/quentinblampey/sopa:2.1.11-baysor>

,

<http://docker.io/quentinblampey/sopa:2.1.11-cellpose|docker.io/quentinblampey/sopa:2.1.11-cellpose>

,

<http://docker.io/quentinblampey/sopa:2.1.11-comseg|docker.io/quentinblampey/sopa:2.1.11-comseg>

,

<http://docker.io/quentinblampey/sopa:2.1.11-proseg|docker.io/quentinblampey/sopa:2.1.11-proseg>

,

<http://docker.io/quentinblampey/sopa:2.1.11-stardist|docker.io/quentinblampey/sopa:2.1.11-stardist>

,

<http://docker.io/quentinblampey/sopa:2.1.11-tangram|docker.io/quentinblampey/sopa:2.1.11-tangram>

@Nicolas Vannieuwkerke you know more about these I believe, care to comment?

Yes, @Quentin Blampey could you please confirm why they need to be custom containers and why it is not possible to have the tools in those containers on bioconda?

Here, these are all official images for Sopa, but with different extra dependency (depending on the suffix) Sopa itself is already on bioconda and biocontainers, but it doesn't allow to install all these extra dependencies (I'm not totally sure I understood/answered your question correctly)

Can't the dependencies be handled via bioconda?

As far as I understand, we can't install extra dependencies through conda, no? E.g., there is no equivalent of

pip install sopa[baysor]

, right? Also, some images require more complex installations, e.g.

sopa-proseg

needs cargo to install proseg (rust) How would you do handle all these extra through a single package in bioconda?

You can install pip packages via conda 🙂. Is

sopa-proseg

available on bioconda? Otherwise it might be worth it to add it

for example: https://github.com/nf-core/modules/blob/master/modules/nf-core/qcatch/environment.yml

I’m not sure to understand - you suggest that I should release one package on bioconda per extra dependency? This would make a lot of packages to publish, and it’s not consistent with what pip does (having a package called

sopa-proseg

seems quite ugly, compared to having a single package called sopa on which we add extra deps)

his would make a lot of packages to publish

That's true, but at the same time like the others said reduces downstream maintainance burden (bioconda has more automation), and keeps your concept of reducing heavy installation requirements

I already have all the Docker images deployed in my own CI for each release, it’s all automated. I have one pip package on PyPI, called

sopa

, which comes with extra, and that we can install with conda, and I make Docker images available for each extra for conveniency. I expect a mirroring to be easier than releasing 6 new packages on bioconda, no ? It would also involve having to maintain some rust/julia dependencies inside these conda yaml files, and I don’t know how easy it is

I expect a mirroring to be easier than releasing 6 new packages on bioconda, no

Not necessarily..

It would also involve having to maintain some rust/julia dependencies inside these conda yaml files, and I don’t know how easy it is

This might sound scary, but also you will also get much more visibility and popularity of the tool if ti's searchable via bioconda

Yeah really, it's a one time commitment that pays off in the future 🙂 That's why almost all pipelines are fully built using packages available in conda

Sopa is already in bioconda, but only the core package, not the extra It seems to be it’s quite an anti-pattern to have one package per extra deps, no? It’s seems more natural to just have docker images on the side, and it’s super easy to handle all the julia/rust stuff

You just need to add those dependencies to conda and put them together in a container using seqera containers 😛

It seems to be it’s quite an anti-pattern to have one package per extra deps, no?

no! There are lots of examples of similar things, e.g. when someone made a (massive) container with a databsae inside, so people madea

-lite

version without the database, for example

You just need to add those dependencies to conda and put them together in a container using seqera containers

But I’ll not be able to do that for proseg and baysor, which uses julia/rust

But I’ll not be able to do that for proseg and baysor, which uses julia/rust

Why not? Plenty of julia and rust tools are on bioconda, you could search for examples in the bioconda recipes repo

Is it your first time adding software to bioconda? If it is, I wrote a bunch of tutorials: https://bioconda.github.io/tutorials/index.html

That was you?

Not the rust specific guidance

Thanks, is there the same for julia? I saw some people already tried to do it for baysor, but couldn’t make it work

Hum so that’s easier…

It's an old recipe though

I get the goal of standardizing via Bioconda/BioContainers, but for sopa this would mean duplicating the packaging into ~7 recipes, with added complexity from Rust/Julia deps… We already provide versioned, tested Docker images per extra, all pushed automatically within the CI (for each release). Rebuilding them via Bioconda would increase maintenance burden and risk divergence from working setups

We recognise it is an annoying part of bioinformatics (many of us also doing the same), however it is an explicit recommendation in the pipeline specifications: https://nf-co.re/docs/specifications/pipelines/recommendations/bioconda So we will push for this at least on a 'best effort basis', and relax it if it is really not possible to put it on conda (e.g. 99% cases of this has been because it's proprietary software or that the conda build just did not support that language). It's important ofr our pipelines to be accessible, and in some cases people don't hace access to docker or even singularity (shocking I know) on their machines, and arguabily bioconda is the defacto standard of bioinformatic tool distribution. So if it really is impossible to get the python/julia deps to work, even after help from the nf-core/bioconda community ,then we can allow a docker-only pipeline

I understand you don’t want to maintain “custom” containers, but is it not possible to just give me the rights to push to an nf-core location? For me, it seems natural that, as a pipeline maintainer, I’m the one responsible for publishing my images . Having access to seqera containers / docker-nf-core or anything similar would be super easy both for me and for you

(sorry @Quentin Blampey had a meeting then get my daughter - will reply tomorrow, but the spacersbger thing might be a count in your favour;) )

To echo James, we know it's a bit painful when starting out, but the benefits down the line are much bigger, so unless there it is technically truly not possible or restricted by licenses we should opt for either bioconda recipes that contain all the individual dependencies or mulled containers that contain sopa + the additional dependencies. With Seqera Containers you can try building those in this interface here by setting sopa and anything you are missing. Can you check which of the process dependencies can be solved with mulled containers and which would need additional packages added to bioconda?

Long thread and I only skimmed it, but hang on - why does it need to be a single bioconda package?

It doesn't

As long as it's installable via a conda

environment.yml

it should be fine, right?

If it's coming from bioconda -> biocontainers though, which is the argument

But biocontainers isn't a requirement, now that we're moving to Seqera Containers

Where does that go though?

Goes into the module, can generate seqera container images from that using the Wave CLI (which is what the automation will do that we're building)

IT's unclear how to skip bioconda to get things on seqera containers

That's where I was getting at with pip

BUt didn't wave have some license restriction or something somewhere...? 🤔

If it's possible to do just an

env.yaml

+ seqera containers, it would be great for me

Maybe I missed something in the thread, but it was all available via conda or pip, right?

(or am I getting confused)

For spaceranger_count, I think it's some license issue, since it's a software from 10X genomics

I'm not sure what you mean about the license

Wave needed a license to use or something

Nope

in some contexts

Maybe you're thinking of the fact that Seqera Containers (which uses wave) doesn't do custom

Dockerfiles

?

Ahhhhhhhhhhhh

It's limited to conda + pip only

That could be it

Maybe I missed something in the thread, but it was all available via conda or pip, right?

Yes, except two extra binaries (written in Julia and Rust), which need to be installed

What are the binaries? Can they be put onto bioconda?

Baysor and Proseg I saw this about rust-proseg on bioconda, but need to check that it works For baysor, I saw some people tried but didn't succeed

If it has an open license, yes

Yeah that's totally fine

I normally leave a 'I will/did this' message on the orginal repo, but never had anyone say NO DONT DO THAT

If you look at bioconda package publishing, you'll see that a few people (many of whom are in nf-core) are incredibly prolific 😅

It's also the same guy that said he failed to do it, see here

I'm like, 80% confident it should be fine. Baysor has a built binary attached to it's GitHub release, so worst case the bioconda recipe can just copy that binary (no build required)

But the binary is only for linux - for macos/windows we have to install it as a Julia package

Having a requirement for only Linux is fine

Yeah but the docker images are fine already, only the conda usage is broken

Something like 98% of nf-core package usage is linux, for context

And how much usage is conda?😅

More than 2%.. 😅 But "support conda" isn't only for people actually using conda. It's also because of our automated build stack

I understand, it just feel weird to have to make a bioconda recipe to build images automatically, while they are already built automatically trough my CI (and versioned accordingly) It feels a bit overkill and a lot of work to do the same in the end Still, I agree that a "personal docker hub" is not a good idea, and I'm happy to move it somewhere else

Well if you just copy the binaries there's no build. It's just leveraging that build system to distribute the binaries via anaconda's cloud hosting basically

But I have to do that for the 7 recipes, right? And then add the corresponding images to the seqera containers? (I also don't know how)

Seqera Containers pulls automagically from bioconda/conda-forge and pip. So once you have the recipes in, the images become available without any additional steps

Just to clarify, should I do: 1. make only one sopa recipe + another recipe with baysor/proseg/etc. then, somehow "merge" them into a new "sopa+baysor" seqera container + "sopa-proseg" container + etc. 2. or, make one recipe for every combination (sopa-baysor, sopa-proseg, etc) and then use the image from biocontainers?

Forget about combining stuff. Just recipes for the individual components.

Because having a recipe for baysor alone is not sufficient, I need sopa (python) + baysor (julia)

As long as you have recipes for sopa, baysor, proseg, you should be fine

But then how can I access an image combining both sopa + baysor? Biocontainer will generate one image for baysor, and one for sopa, right?

Not using Biocontainers

Sorry I'm confused 😅

This thread is overly confusing 😅

I'll try to clarify - Let me know if this is correct: I need to fix two things right now: 1. conda support. This is only possible for non-Visium-HD technology, because spaceranger_count doesn't support conda anyway. I need to update the

environment.yml

files, should be relatively easy 2. docker images: I need to move from my current images to some images hosted on seqera. How to do that? Is it possible to just mirror my images, or do I need to do something more complex? Knowing that I'll need to combine multiple recipes (for instance sopa + baysor)

To generate images, easiest is to use the Wave CLI:

brew install seqeralabs/tap/wave-cli
wave --conda-file=environment.yml --freeze
wave --conda-file=environment.yml --singularity --freeze

That'll give you docker and singularity image URIs

Okay, thanks for the clarifications! Last question to be sure I understand: for sopa+baysor, I need to make a recipe for baysor alone, and then inside my nf-core/sopa pipeline I create a

environment.yaml

which installs both sopa and baysor, correct?

Correct 👍🏻

Hey @Phil Ewels maybe some docs on this would be helpful 😉

Once we finish #C09V4RAHV0X there will be. Also a bunch of new CLI tooling to do all this stuff for you.

Not yet

Hi, quick update: after many trial and error, I was able to make the PR for the baysor recipe and the tests are finally passing (here) I'm wainting for it to be merged, but meanwhile I was looking at how to effectively create the containers on seqera You mentioned the Wave CLI, but I also see that we can directly create them via the seqera UI. I was wondering is there is any difference between both (except the Wave CLI being more automated, of course)?

Added a couple of review comments approved with comments

Thanks for your comments @Phil Ewels I had a working version (after your comments) and then got a comment from another reviewer suggesting to build from source instead of downloading the zip - see this PR I tried but couldn't find a solution yet (I don't know a lot about Julia and the internal of bioconda-recipes, so I struggle a lot), it's taking much more time than expected, do you have any ideas to help fixing and merging this?

I'll point Claude at it now and see if I can make any progress