Are we ok to remove `.gitpod.yml` config files fro...
# dev-environments
p
Are we ok to remove
.gitpod.yml
config files from the pipeline template / core repos now? 🤔
m
YES
We already moved out of gitpod for training
m
m
Not yet
m
We should do it together with the migration to devcontainers in my opinion. The container needs to be corrected in particular
party this 3
p
Yup, fair point 👍🏻
I've been working quite a bit on the devcontainers configuration for training.nextflow.io - can hopefully lift that essentially as-is
🙏 1
Would be good to get some eyes on it / testing though
m
Where's the Dockerfile?
p
There isn't one 😁
It builds off
<http://mcr.microsoft.com/devcontainers/base:ubuntu|mcr.microsoft.com/devcontainers/base:ubuntu>
and then uses Devcontainers features instead
m
Then it's likely missing what we're using like apptainer and docker
p
Nope, docker is here, apptainer is here
The final built image is still pushed in this action to here for faster boot times, so that it doesn't have to rebuild all the time
m
OK. I'll have to look over it later. I've a bunch of meetings from 13.00
👍🏻 1
p
The one part we haven't fully resolved is getting it to work well both locally and on codespaces. @Jonathan Manning has been working on that and I owe him a review on this PR to resolve that.
We had a Dockerfile before, with a few features to add stuff like docker-outside-docker, but the build time was very slow and the environments kept running out of space and borking on codespaces.
I ended up doing a fairly deep-dive on analysing docker image layer size and build process, and the above is what I came up with. Using features consistently for everything, rather than a mixture, seems to be quite a bit faster for codespaces boot time. And the combination with that base image gives plenty of head room on the available disk space for build.
Definitely keen to hear what others think though. Especially if anyone can figure out how to resolve local + codespaces into a single devcontainer setup, something that we haven't quite been able to manage yet.
j
> Definitely keen to hear what others think though. Especially if anyone can figure out how to resolve local + codespaces into a single devcontainer setup, something that we haven't quite been able to manage yet. After a few separate rabit-hole attempts I convinced myself this wasn't possible due to mostly to the docker-outside-docker. You have to make the local paths valid both on the main container and any Nexflow workers, and than involves using variables specific to the local devcontainers setup. Happy for a smarty-pants to prove me wrong though, or maybe we do docker-in-docker (I don't remember the case against that)
f
perhaps this is not relevant on this thread but during the hackathon I’ve noticed Docker not working on code spaces while working fine on gitpod. have the feeling (also using solely code spaces in class) it’s still not very stable. just my own feeling? the consequence has been I couldn’t test all profiles on github while I could on gitpod
p
Could you elaborate? We had problems with Codespaces builds failing on some earlier versions but I hadn't heard of any problems since the 2.1.4 release..
(2 weeks ago)
f
sure sorry. the issue is with pipeline template v.3.2.0. (or any update to codespaces, can’t say). when I load a codespace from an nf-core pipeline repo created with nf-core tools v.3.2.0, I can’t run an nf-test with
--profile docker
because the docker daemon is not working. checking
docker ps
gives the message
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
Didn’t have an easy fix. I suppose it’s got to do with the docker in docker thingy, but I haven’t experienced this before. In order to answer your message I’ve double checked with the
.devcontainer
used in the nf-core modules repo, and it’s different from the one shipped with the pipeline template, in particular the modules one has these features:
Copy code
"features": {
        "<http://ghcr.io/devcontainers/features/github-cli:1.0.11|ghcr.io/devcontainers/features/github-cli:1.0.11>": {
            "digest": "sha256:464564228ccdd6028f01f8a62a3cfbaf76e9ba7953b29ac0e53ba2c262604312"
        },
        "<http://ghcr.io/devcontainers/features/docker-in-docker:2.9.2|ghcr.io/devcontainers/features/docker-in-docker:2.9.2>": {
            "digest": "sha256:9074d8eaf4585ab1498c81a8762078992725614fc7a9cfafe799aa22717a6bbf"
        }
    }
and when spinning up a codespace with this one, the docker daemon works. I’ve tested it by adding these to the
devcontainer.json
in the nf-core pipeline’s fork, and the docker daemon works. so it might be perhaps be an easy fix to the next template release? and/or inform pipeline developers in the meanwhile. this is only important to run nf-tests when developing local modules or local workflows in a pipeline I suppose, but it prevents from testing them with docker. I’d be quite in favour of running all conda, docker, singularity before making a pull request, even if it’s local modules. I didn’t notice this difference in the
devcontainer.json
definition until now, so during the hackathon I’ve developed in gitpod where the problem doesn’t appear. hence my message here.
p
ohhh you're not talking about the training.nextflow.io codespaces devcontainer, you're talking about launching a codespaces from the pipeline devcontainers config?
✅ 1
Ok then yes, further evidence that we need to give some love to the devcontainers config in the pipeline template, I agree 👍🏻
I've commented on https://github.com/nf-core/tools/issues/3466 and added it to the milestone for the next release, so that we can hopefully make sure that it's included in the next pipeline update sync.
🚀 1
m
Something else to consider for the Devcontainer documentation: https://nfcore.slack.com/archives/CE5LG7WMB/p1743781922533119
🤔 1