https://bentoml.com logo
Join Slack
Powered by
This message was deleted.
# ask-for-help
s
This message was deleted.
e
Here is the command I'm running:
Copy code
+ DOCKER_BUILDKIT=1
+ bentoml containerize keras-ctr-service:latest --docker-image-tag keras-ctr-service:latest
Here is the error I'm getting:
Copy code
WARNING: You are using pip version 22.0.4; however, version 22.3 is available.
You should consider upgrading via the '/opt/atlassian/pipelines/agent/build/venv-containerize-service/bin/python -m pip install --upgrade pip' command.
Error: [bentoml-cli] `containerize` failed: BentoML requires BuildKit (via Docker Buildx) to be installed to support multi-arch builds. Buildx comes with Docker Desktop, but one can also install it manually by following instructions via <https://docs.docker.com/buildx/working-with-buildx/#install>.
error: Recipe `containerize-bento-service` failed with exit code 1
error: Recipe `containerize-bento-service-cd` failed with exit code 1
I don't know much about "buildkit", but I followed the Bitbucket pipelines documentation here to enable it: https://bitbucket.org/blog/announcing-support-for-docker-buildkit-in-bitbucket-pipelines, AKA, I added 
DOCKER_BUILDKIT=1
to the 
bentoml containerize ...
command, and I included the 
docker
service.
^^^ Here is an equivalent 
bitbucket-pipelines.yml
to what I'm running
I noticed that the 
bento-cli
runs 
docker buildx --help
to check whether 
buildkit
is installed. It asserts that the string 
"buildx"
is in the output. We can see that it indeed is not in the output in bitbucket pipelines.
For a second there, I thought I had it. I manually installed 
buildx
into the docker plugins directory in bitbucket by following the 
docker/buildx
installation guide. So I added this step to my pipeline file:
Copy code
- |
              export BUILDKIT_FPATH=$HOME/.docker/cli-plugins/docker-buildx
              mkdir -p /root/.docker/cli-plugins
              wget <https://github.com/docker/buildx/releases/download/v0.9.1/buildx-v0.9.1.linux-amd64> -O $BUILDKIT_FPATH
              chmod +x $BUILDKIT_FPATH
But I got a new error instead. It looks like docker build is somehow trying to build the container with 
--privileged true
.
Copy code
+ npx just containerize-bento-service-cd
/tmp/justGgW4wU/containerize-bento-service-cd: line 126: deactivate: command not found
not in venv
WARNING: You are using pip version 22.0.4; however, version 22.3 is available.
You should consider upgrading via the '/opt/atlassian/pipelines/agent/build/venv-containerize-service/bin/python -m pip install --upgrade pip' command.
+ DOCKER_BUILDKIT=1
+ bentoml containerize keras-ctr-service:latest --docker-image-tag keras-ctr-service:latest
Building docker image for Bento(tag="keras-ctr-service:htaqimcwtsdanvxg")...
#1 [internal] booting buildkit
#1 pulling image moby/buildkit:buildx-stable-1
#1 pulling image moby/buildkit:buildx-stable-1 3.9s done
#1 creating container buildx_buildkit_default 0.0s done
#1 ERROR: Error response from daemon: authorization denied by plugin pipelines: --privileged=true is not allowed
------
 > [internal] booting buildkit:
------
ERROR: Error response from daemon: authorization denied by plugin pipelines: --privileged=true is not allowed
Failed building docker image: Command '['docker', 'buildx', 'build', '--progress', 'auto', '--tag', 'keras-ctr-service:latest', '--file', 'env/docker/Dockerfile', '--load', '.']' returned non-zero exit status 1.
error: Recipe `containerize-bento-service` failed with exit code 1
error: Recipe `containerize-bento-service-cd` failed with exit code 1
Bitbucket pipelines really seems not to like BentoML 😞
Omg, I think this is official bitbucket documentation saying 
docker buildx
is intentionally restriccted: https://support.atlassian.com/bitbucket-cloud/docs/run-docker-commands-in-bitbucket-pipelines/#Docker-BuildKit-restrictions
😒 1
I think Bitbucket Pipelines may truly be incompatible with 
bentoml containerize
which I imagine is a big deal for unfortunate companies like us who use Bitbucket and Bitbucket Pipelines for CI/CD, including the building of all production docker images.
a
Since we don’t yet have a solid CI story, I believe that we might need to think about our usage for buildx. We are currently building container image via docker buildx (which is the CLI plugin from docker that has BuildKit enabled.)
I believe you can use buildkit with default 
docker build
by passing 
DOCKER_BUILDKIT=1
. I believe that we can still use buildkit without having to use 
docker buildx
. In terms of CI story, buildx only fully support GitHub Actions 😐
s
BentoML should provide a generic Dockerfile that does not make assumption on the build tool. In BentoML 0.13, this would not be an issue at all. Now, BentoML 1.0 adoption is more difficult because of these assumptions.
e
Can you think of a possible workaround for this? Like, could I manually purge the Dockerfile of any buildkit-dependent syntax, for example? (Ideally there'd be an easier way than that πŸ˜„ )
s
Cc @Tim Liu
t
Thanks for the ping @Shihgian Lee! Another case where we're running into issues with buildx. @Aaron Pham Maybe we should talk about what it may take to have a simple way to use build, perhaps with certain limitations or a way to escape hatch into a more technical implementation if needed
@Eric Riddoch As I understand from your link, bitbucket pipelines does not support buildx, but still does support buildkit. We think that you can bypass the need for buildx by using 
DOCKER_BUILDKIT=1
@Aaron Pham Does that seem right? (Given Eric's observation that we may use docker buildx to check if buildkit is present?) Just FYI for everyone, looks like Bitbucket doesn't support buildx because of this bug as noted here All that said, we need to make the experience better out of the box especially in cases where you may not have buildx/buildkit
πŸ‘ 2
e
I tried 😞. If you look, the screenshots show a few different ways of an attempt to export that variable.
πŸ‘€ 1
a
we are currently hardcode buildx. I’m working on fixing that.
e
^^^ Incidentally, I was able to get the string 
"buildx"
to appear in the 
docker buildx --help
output. I mean, it was an error message saying it was an invalid plugin, but still πŸ˜…. That bypassed bento-cli's check for that string and allowed the build to proceed. But then I got the "elevated privileges" error and gave up.
Nice result: I'm doing a POC of BentoML right now, so I figured I might as well throw in an AWS CodeBuild POC while I'm at it. It seems to be going well. Buildkit is working just fine. Sad news for me: At BEN (where I work) currently, CodeBuild is kind a 2nd class build tool in the sense that if my team chooses to go down this road, we lose support from other DevOps folks in the company. So it's hard to say, but based in this, we may need to make a specialized IaC library that makes CodeBuild easy for our data scientists to use while Bitbucket Pipelines still struggles with buildx
s
The biggest value of BentoML is its ability for us to build and deploy to existing infrastructure. It is sad to hear that BentoML 1.0 gives you so much trouble. It is a departure from BentoML 0.13. I agree with @Tim Liu that BentoML 1.0 should be fixed to make build and deployment easier for existing infrastructure.
t
0.13 had its issues too :) but great users like you guys helped us work them out. The team is working on the buildx issue right now
πŸ™ 1
We think we have a fix that deprecates the need for buildx here: https://github.com/bentoml/BentoML/pull/3160/files Still testing as a couple other projects depended on it, but we think this is will solve your issue with bitbucket @Eric Riddoch
e
Whoa that would be great! Taking a look now
What would be the best way to try this? Can I pin an unofficial version with 
pip install ...
and then try to use this new bento version when I run 
bentoml containerize
?
a
We will try to get this merge into main, and then after that you can do
Copy code
pip install git+<https://github.com/bentoml/bentoml@main>
πŸ‘ 1
e
@Aaron Pham @Tim Liu I saw that the PR that might fix the build for bitbucket pipelines got closed in favor of this one: https://github.com/bentoml/BentoML/pull/3164 Could this ^^^ be merged? I see that a failed check is due to open conversations. None of those conversations seem to be about issues. Only comments.
a
We r in the process of reviewing it
πŸ™Œ 2
Hi @Eric Riddoch and @Shihgian Lee The builder PR has now been merged. Feel free to try it out and would love to hear feedback.
πŸ™ 1
e
Yeeeees! If not next week, I'll definitely have tried this the week after.
Thanksgiving and all that πŸ˜„
a
have a good weekend πŸ™‚
e
@Aaron Pham I finally got around to trying the non-buildkit build feature. I seem to be getting the same error as last time: Command
Copy code
python -m venv venv-containerize-service
. ./venv-containerize-service/bin/activate \
&& python -m pip install bentoml>=1.0.10 \
&& which bentoml \
&& python -m bentoml --version \
&& DOCKER_BUILDKIT=0 \
	python -m bentoml containerize keras-ctr-service:latest --docker-image-tag keras-ctr-service:latest --platform linux/amd64
Logs
Copy code
version 1.0.10
Error: [bentoml-cli] `containerize` failed: BentoML requires BuildKit (via Docker Buildx) to be installed to support multi-arch builds. Buildx comes with Docker Desktop, but one can also install it manually by following instructions via <https://docs.docker.com/buildx/working-with-buildx/#install>.
Can you tell if I'm doing something obviously wrong?
a
this log comes from older version of bentoml. The container API are merged on main and not yet released. You probably need to install bentoml with
Copy code
pip install git+<https://github.com/bentoml/bentoml.git@main>
We are in the process of releasing 1.0.11. And it will include the container change.
python -m
might not respect the venv
e
Ah, okay!
I'll try that pip install statement right now
a
you can try 
pip install bentoml>=1.0.11
2 Views
Open in Slack
PreviousNext
Powered by