Hi folks! We've been having issues with using con...
# pipeline-maintainers
j
Hi folks! We've been having issues with using conda packages with nf-core/pixelator - mostly relating to some of the packages we depend on in turn not being updated and having very complicated build processes that make it almost impossible for us to go in ourselves and support the updates. For this reason we are thinking about dropping conda support in the nf-core/pixelator pipeline and provide our own custom containers that we know work and that we can much more easily provide support for. I saw a discussion on a similar topic here, which seemed to indicate that conda is recommended but not mandatory. I just wanted to check in here and see if that is still the community consensus.
j
Depends on the levels of conda issues, conda support is strongly recommended even though it's not mandatory
Can you describe them?
j
The specific issue right now is that the release of the latest version of this conda forge package is blocking us from releasing an update of our pixelator bioconda package. We are unfortunately not able to wait for that to be merged or allocate resources towards solving it ourselves at this time. Our customers rely on the pipeline in order to be able to process the data that they generate using our assays. We have recently released a new type of assay (the Proximity Network Assay) which in turn requires a lot of changes to the pipeline. These are already working on our docker containers - and ideally we would now like to update our bioconda/pixelator package and use that for the conda environments and the associated biocontainers docker/singularity containers. However, due to the issue with the conda package linked above we are unable to do so, and that is holding up the entire process. What I would like to do is to temporarily remove conda support from the pipeline in order to ensure that those customers that are able to run containers on their infrastructure have to option to do so, until we are able to roll out later with full conda support once all our dependencies have caught up. I guess we are in a bit of an unusual situation in that we maintain the full system, from the assay, to the software package, to the pipeline, and being a company it is critical to us that our customers have some solution for processing the data (even if that solution is less than ideal). I think we are still trying to figure out a little bit of how that makes us fit into the nf-core ecosystem, and I would be happy to discuss that industry perspective on these things further if there are folks that are in a similar situation.
j
Sorry, you replied during school run hours 😅 I had a very quick look at your pipeline - is it correct that the vast majority of your modules are using just one container/tool (pixelator?), in which case yes I think it makes sense to (hopefully) temporarily stop support for
conda
. I was going to say if it was just for a specific 'end point' tool maybe can just block conda support for that module, but I can see why this is quite catastrophic. So from my PoV this would be fine. I guess the question is more how the removal of support should be implemented. If you expect the conda issue to be resolved in the near future you don't want to do something too drastic. Maybe use the internal nextflow
conda.enabled
variable to add a check that if it's
true
throw an error? What do you think <!subteam^S016E36N7TQ> <!subteam^S044DU0NG68>?
this 2
p
Yeah, only alternative that I can think of is to repackage the conda tool yourself in a custom conda channel. But that's a bit of a faff and not ideal. Seems like it's a huge package (Facebook?) so not easy to influence maintainers. Fine by me to drop conda for now. Note that custom containers should still be on the nf-core quay.io account and the dockerfiles should be included alongside the module so that others can maintain and push new images if needed. Someone from the core team can push the image - either building ourselves, or doing a docker pull + push from another docker registry that you tell us about.
👍 1
j
Hard to say how long it will take before it is resolved. But I think your solution of looking for the
conda.enabled
variable and throwing an error if that is true sounds like a good work around for now. @Phil Ewels we have taken up setting up and maintaining conda packages for other of our dependencies previously, but as you say, this one is just to big and complicated for us to take on. When you say the docker files should be available along side the module, where does that mean we should place them? We keep our public containers here: https://github.com/PixelgenTechnologies/pixelator/pkgs/container/pixelator and the docker files and docker-bake file are in the pixelator repo: https://github.com/PixelgenTechnologies/pixelator/tree/dev
👍 1
And thank you both you for taking the time to understand and work with us to resolve this issue.
p
If the Docker image is for the
pixelator
CLI specifically, then I think that's fine. If the docker image is for a step in the pipeline which is pipeline-specific (eg. includes multiple tools etc), then the module-specific Dockerfile should be kept in the module directory basically. And the image pushed to quay.io/nf-core
But if the image is just for the pixelator tool only then it's fine to keep that separate and use a 3rd party docker image I think. That's done elsewhere for proprietary software already. Bit of a last-resort, but doable if no other way.
j
Yeah, it is only for the
pixelator
tool. I think for us the easiest way to do this would be to keep the containers on github. I think from the community point of view, since we have everything under and open-source license it should be possible for anyone to audit how the container is built and if someone wishes to fork it and build their own containers they are of course welcome to do so. We do need to iterate rather quickly on these things. Moving forward I hope to start doing at least one release of pixelator each month, so if we need to hand that work of to someone else it will create quite a lot of extra work for us and other, that I would like to avoid.
👍🏻 1
👍 1
p
Sounds good!
j
@Johan Dahlberg I just remembered we have tocheck if conda is enabled for a module
`println(session.config) Gives me
Copy code
]], conda:[enabled:true, channels:[conda-forge, bioconda]], shifter:[enabled:false], prov:[enabled:true, overwrite:true, params:[outdir:./results], file:./results/manifest.json], session:[:], executor:[:], runName:golden_goodall, workDir:work, poolSize:16]
Maybe you can use
session.config.conda.enabled
?
j
Nice. I can try that.