How does BentoML charge after deployment? Does the billing start as soon as it is deployed (for GPU and CPU), or is there no charge if it is not in use after deployment, with billing only starting when GPU or CPU resources are utilized?

Billing is based on node hours of the deployment

You can configure a deployment to scale to zero which will be no cost to the user when it has scaled down

Thank you, I will take a look at the documents related to scaling.

Do you have some stats on cold start times on bento cloud? My models are quite small, ~ 1.2 GB.

@Matěj Šmíd From what I understand, you shouldn’t worry too much about cold start time (though it might be good to confirm with @Chaoyu). You don’t need to be concerned about the size of the model, as it won’t affect the cold start, see: https://docs.bentoml.com/en/latest/build-with-bentoml/model-loading-and-management.html You can store the model in BentoML storage, and each time it starts, BentoML can mount it to reduce the time needed to download the model each time.

We want to start with scaling to zero. I expected there will be some cold start time for spinning up the first instance.

We want to start with scaling to zero. I expected there will be some cold start time for spinning up the first instance.

I checked and found that it’s possible to upload the model to BentoML storage before starting (0 instances). To be honest, I haven’t actually started running it yet, so I can’t confirm the final cold start time. However, I plan to run actual tests today or tomorrow.

Please keep me posted, I'm definitely interested.

@Matěj Šmíd 30s ~ 95s