https://bentoml.com logo
Join Slack
Powered by
Using new bentoml way to define the service using ...
# ask-for-help
t
Using new bentoml way to define the service using class decorator. I have 4 gpus in instance, but bentoml does not use all 4 of them, how to force bentoml to run one instance of runner on each of them ?
t
@Jian Shen Yap
j
Hello, are you deploying this in your own instance?
A few pointers here: 1. the resources configuration is for BentoCloud configuration. if you are deploying locally or in your own instance, that config will be ignored. 2. to utilize multiple cards, you will have to write your own logic to port them into different machine
t
The previous way of writing runners was compatible with this use case
j
could you show the example how would you previous write your runners?
t
Like custom runner YOLO example in bento ml GitHub examples repo
j
were you following this example?
t
image.png
if i run this example, each of the 4 GPUs gets an instance of the model (on instance with 4 cpu cores). That was very helpful functionality
new service decorator method lacks this functionality.
j
Thanks for spotting this, we will get back to you!
👀 1
t
image.png
Hi @Jian Shen Yap any update on this ?
f
In BentoML 1.2 we always make all requested GPUs visible to all workers. One would need to assign individual GPUs themselves based on the worker ID. that info can be obtained from 
bentoml._internal.context.component_context.component_index
for now but we will make it accessible from the top namespace.
❤️ 2
m
@Frost Ming can you please elaborate on how exactly one should go about assigning GPU to workers? Right now I am using 10 workers but they get loaded in CPU instead of GPU. @Talha Yousuf please guide me if you were able to make it work
f
Copy code
def __init__(self):
    assigned_gpus = os.getenv("CUDA_VISIBLE_DEVICES", "")
    if not assigned_gpus:
        # cpu logic
    else:
        os.environ["CUDA_VISIBLE_DEVICES"] = assigned_gpus.split(",")[component_context.component_index]
Open in Slack
PreviousNext
Powered by