Apache Pinot

image.png

Hello, I noticed Apache Pinot container images have a target CPU instruction set encoded in image versions. Why is that?
I usually see multi-platform images being used. Is there anything special about Pinot images? Or it’s just the way the Apache release infrastructure is set?

arm64 is majorly used for people trying out pinot with macbook M-series CPU

Also AWS Graviton machines are using arm64

hi Xiang, I understand that. I use ARM mostly on a laptop too. but why is it a separate version instead of a multi-platform image? doesnt a CPU hard-coded in a version complicate deployments, user tests, etc?

The issue is that multi-arch docker image built usually failed/timeout from our docker image publish pipeline.

right, ok, so it’s just a build infra setup. I saw it and I thought that perhaps there was a deeper reason and it made me curious. thanks a lot for the insight!

not really, we use github action (free resource) to build publish image. The instance cannot finish dual platform docker image built within 6 hours, hence we built them separately. Once we find a way (also free), we will publish the multi-arch image.

This is a typical time for our nightly builds of amd64/arm64/dual

the build is on amd64 arch instance, you can see that arm64 build is significantly slower

<@UGRJA9TEH> a Docker manifest could be used to allow for building each image in parallel, then “combining” them by creating a manifest and pushing that for a given tag. This would allow the host system to select the “right” version of the image from the manifest based on host architecture, but using the same image tag. The approach is described as “the hard way” here: 

<https://www.docker.com/blog/multi-arch-build-and-images-the-simple-way/|https://www.docker.com/blog/multi-arch-build-and-images-the-simple-way/>

This way, each arch build can be done separately as their own CI job, so hopefully timeouts wouldn’t happen (assuming that each individual arch build does not time out already)

Yeah, last time I checked, the GitHub docker doesn’t support this. I will check it again. 

Thanks <@U025EKQUZU7> I’ve made the docker images separately and use docker manifest to assemble them.
Previously I was using `docker buildx build`, which should build multi-arch image, but due to 6 hours CI time limit, I split them into two images. Docker manifest cannot create on top of docker buildx images, so I need to use `docker build` for each platform then assemble them.
More ref: <https://github.com/apachepinot/pinot-fork/blob/master/.github/workflows/build-multi-arch-pinot-docker-image.yml>
<https://github.com/apachepinot/pinot-fork/blob/master/.github/workflows/scripts/docker/.pinot_single_platform_docker_image_build.sh>
<https://github.com/apachepinot/pinot-fork/blob/master/.github/workflows/scripts/docker/.pinot_multi_arch_docker_image_manifest_package.sh>

Also due to end of life of JDK 11, we switch the base JDK default from `openJDK` to `amazoncorretto` , meanwhile also publishing `ms-opnejdk` for Azure users.

<@UGRJA9TEH> I do see that the amazoncorretto  image is available for latest release in docker but that release is still snapshot right ? When will the 0.13 release be available or are well planning a point release ?

I think we will start 0.13.0 from April/May. It will use new image.