This message was deleted BentoML #ask-for-help

Join Slack

This message was deleted.

# ask-for-help

Slackbot

11/18/2022, 9:35 AM

This message was deleted.

Sean

11/18/2022, 8:30 PM

In GPU enabled node group setup, we had success using the

Amazon Linux 2 GPU Enabled (AL2_x86_64_GPU)

image. To recognize GPU resources in the cluster, we recommend installing the Nvidia k8s-device-plugin. Once the plugin is installed, you can specify

<http://nvidia.com/gpu|nvidia.com/gpu>

custom resources from Yatai and the

BentoDeployment

CRD. See more, https://docs.bentoml.org/projects/yatai/en/latest/advanced_guides/gpu_deployment.html

Sean

11/18/2022, 8:32 PM

cuda

can be specified in the Docker requirements of the bento. See an example here, https://github.com/bentoml/stable-diffusion-bentoml/blob/main/fp16/bentofile.yaml

👀 2

Boris Bibic

11/23/2022, 8:28 AM

Hi Sean, thanks for the info. We already installed the proposed Nvidia driver, but we’ll look into the

cuda

inside the Docker requirements of the Bento (@Rustem Salawat).

Rustem Salawat

11/23/2022, 8:53 AM

@Boris Bibic yes we have cuda inside

Copy code

service: "service.py:svc"
include:
- "service.py"
docker:
    cuda_version: "11.6.2"
    dockerfile_template: ../Dockerfile.template
python:
  packages:
  - transformers[sentencepiece]
  - torch
  - flax
  - jaxlib

👍 1

Rustem Salawat

11/23/2022, 8:53 AM

Copy code

{% extends bento_base_template %}
{% block SETUP_BENTO_COMPONENTS %
{{ super() }}
{% endblock %}

{% block SETUP_BENTO_ENTRYPOINT %}
RUN cd /usr/bin && ln -s /usr/bin/python3 python

RUN echo $PATH
{{ super() }}
{% endblock %}

Open in Slack

Previous Next