What is the reason for these three equivalent containers being so radically different in size:

1.84 GB community.wave.seqera.io/library/jasminesv 
271.18 MB quay.io/biocontainers/jasminesv
  2.22 GB manual-build-seqera-dockerfile

Over the course of a whole pipeline with many steps this could amount to hundreds of gigabytes of extra container pulls (assuming similar size diffs between all seqera containers and biocontainers. I also don’t understand why manually building the container based on the seqera containers Dockerfile definition from the build page results in a container which is over 20% larger?

You could look what's inside (start a shell with Docker or Apptainer and see with

du

where most of the megabytes are). If the `Dockerfile`s are available, you can see if anything is obviously different between them. One issue with Docker is that could can fairly easily modify the container and delete or add things that may not be listed in the

Dockerfile

. And if the "recipe" for building is not strict on version requirements, your manual build may contain the newest packages for lots of things, which may well be larger than older versions. (I don't even want to think about environment variables, etc., that may influence what gets installed or how things get compiled [e.g. adding or stripping debug info].) I don't know anything about

jasminesv

, but some containers contain data (e.g. reference files), while others don't. Some might have additional tools or OS packages that may be bloat or maybe useful on occasion. Sometimes they do contain things you might consider useless crap, e.g. I just looked at one of our containers with R packages, and it contains no less than 3.5 GB of cache from the package manager

renv

. (Of course, if packages are disappearing from Bioconductor, etc., such things might be needed to faithfully reconstruct the container, but when running it they just take up space.)

So I did that, for the seqera vs manual build I found that the difference appears to be in

/opt/conda/pkgs

but weirdly

du

produces inconsistent results in that folder in the same container. One call it will say something is 400M the next it will be 800M - very weird. For the Seqera vs Biocontainers it seems like there is no real conda pkgs overhead and I guess I'm wondering if the Seqera containers could be more aggressively pruned? A 2GB difference in container size (~10x) is really insane given they both purport to build the same conda pkg

That is very strange -- have you tried to see if

--apparent-size

makes a difference for

du

? There is nothing from the outside mounted to the container while you are doing that, I presume ... If that persists, you could do something like

ls -lR

every few seconds to see if anything changes (and what). I actively try to avoid Conda whenever I can (bad experiences the first time I tried using it -- inability to install stuff because it could not find a way through dependency hell, while wasting hours trying), so I have little experience with it, but I didn't think it had a ton of overhead if you just install one environment/set of packages. I wonder if some containers have stronger compression than others. 🤔

weirdly

du

produces inconsistent results in that folder in the same container. One call it will say something is 400M the next it will be 800M - very weird.

Some filesystems do compression, deduplication, etc, in the background. I’ve had the case where

du

would successively print different sizes, especially if other processes in parallel would

ls

or

cat

the file. That’s because

du

was returning the actual size on disk as reported by the filesystem, Adding

--apparent-size

solved it for my purpose

I will try that. Regardless, I want to understand why Seqera containers are 10x the equivalent biocontainer, especially given we are moving to more of them in nf-core

I think we're defaulting to Pixi now, right? You can check the

Dockerfile

and the other details on the wave build URL

Looks like this is a rather old build - 2025-04, but is uses micromamba:

FROM mambaorg/micromamba:1.5.10-noble
COPY --chown=$MAMBA_USER:$MAMBA_USER conda.yml /tmp/conda.yml
RUN micromamba install -y -n base -f /tmp/conda.yml \
    && micromamba install -y -n base conda-forge::procps-ng \
    && micromamba env export --name base --explicit > environment.lock \
    && echo ">> CONDA_LOCK_START" \
    && cat environment.lock \
    && echo "<< CONDA_LOCK_END" \
    && micromamba clean -a -y
USER root
ENV PATH="$MAMBA_ROOT_PREFIX/bin:$PATH"

Ah ok. Maybe try using the Wave CLI and --template pixi, you might find it's quite a bit smaller

Ok a bit better -

1.51 GB

. I don’t really know enough about how the biocontainers are actually built (and weirdly although the

jasminesv

one is on their registry and definitely exists I can find no entry for it on GitHub) but I do think that the much smaller size makes me a bit cautious about wholesale moving to the Seqera containers.

The bioconda images are built on the

busybox

base image, which is a very bare bones one

Interesting. Do you know if there's any intention or scope with the container work for nf-core to look at slimming down whatever base we use?

Haven't really talked about it before. <!subteam^S08LAB76NBU> what do you think?

Please make an issue in the tools repo. Something we should look into once we have everything in place.