BentoML

Hi Everyone! We are evaluating BentoML to use internally for our organization.Thanks for creating this great product. I wanted to understand the use of yatai-service apart from providing a Web-UI for current model services and deployments. When I fire the bentoml-yatai-service, I see the two servers ( a gRPC one and another a Node one). What is the gRPC server designed to serve? I am sure I am missing something there,please help.

Hi <@U01EATHM5EG> you could use tensorflow artifact to pack your model and do inference in apis.

You can checkout examples for better understanding

can you send me the link to the examples?

Great, love to hear how your adoption goes and if you have any feedback on the batching algorithm 

Our models are built with Keras Functional API and logged in MLFlow, so i stumbled across all the errors mentioned in this ticket. <https://github.com/bentoml/BentoML/issues/1188>

however these are resolved with the workaround as per the issue

when micro batching is enable, i get such error.
```[2020-11-13 17:10:08,117] ERROR - Traceback (most recent call last):
  File "/Users/jianshen/Desktop/Projects/reco-engine-v2/venv/lib/python3.8/site-packages/bentoml/marshal/marshal.py", line 210, in request_dispatcher
    resp = await self.batch_handlers[api_name](req)
  File "/Users/jianshen/Desktop/Projects/reco-engine-v2/venv/lib/python3.8/site-packages/bentoml/marshal/dispatcher.py", line 141, in _func
    raise r
  File "/Users/jianshen/Desktop/Projects/reco-engine-v2/venv/lib/python3.8/site-packages/bentoml/marshal/dispatcher.py", line 201, in outbound_call
    outputs = await self.callback(tuple(d for _, d, _ in inputs_info))
  File "/Users/jianshen/Desktop/Projects/reco-engine-v2/venv/lib/python3.8/site-packages/bentoml/marshal/marshal.py", line 109, in _batch_handler_template
    return await func(requests, api_name)
  File "/Users/jianshen/Desktop/Projects/reco-engine-v2/venv/lib/python3.8/site-packages/bentoml/marshal/marshal.py", line 286, in _batch_handler_template
    return tuple(
  File "/Users/jianshen/Desktop/Projects/reco-engine-v2/venv/lib/python3.8/site-packages/bentoml/marshal/marshal.py", line 287, in &lt;genexpr&gt;
    aiohttp.web.Response(body=i.body, headers=i.headers, status=i.status)
  File "/Users/jianshen/Desktop/Projects/reco-engine-v2/venv/lib/python3.8/site-packages/aiohttp/web_response.py", line 540, in __init__
    super().__init__(status=status, reason=reason, headers=real_headers)
  File "/Users/jianshen/Desktop/Projects/reco-engine-v2/venv/lib/python3.8/site-packages/aiohttp/web_response.py", line 86, in __init__
    self.set_status(status, reason)
  File "/Users/jianshen/Desktop/Projects/reco-engine-v2/venv/lib/python3.8/site-packages/aiohttp/web_response.py", line 119, in set_status
    self._status = int(status)
TypeError: int() argument must be a string, a bytes-like object or a number, not 'NoneType'```

i printed out `i.status` -&gt; it is `None`. For testing purposes I modify is so that it will return 200 when `i.status` is None , and it successfully ran with the right prediction

What would be the possible cause for the api server to return status code of None when using microbatching?

Hi <@U01EKMW9X0A>. I haven't seen a similar problem before. May I take a look at your python script that defines the bento service?

Hi <@UQF2L5GQ0>, thanks for replying.

```import bentoml
import tensorflow as tf
import numpy as np

from bentoml.frameworks.tensorflow import TensorflowSavedModelArtifact
from bentoml.frameworks.keras import KerasModelArtifact
from bentoml.adapters import TfTensorInput
from bentoml.adapters import DataframeInput

import pandas as pd

@bentoml.env(infer_pip_packages=True)
# @bentoml.env(pip_packages=['tensorflow', 'numpy', 'pandas','scikit-learn', 'keras'])
@bentoml.artifacts([TensorflowSavedModelArtifact('model')])
class KerasModelService(bentoml.BentoService):
    @bentoml.api(input=DataframeInput(), mb_max_latency=1000, batch=True)
    def predict(self, df: pd.DataFrame):
        int_oh = np.append(np.zeros((2048 - df.shape[0], df.shape[1])),df,axis=0)
        exp_int_oh = np.expand_dims(int_oh, axis=1)
        batch_preds = self.artifacts.model(exp_int_oh.astype('float32'))
        preds_idx = tf.argsort(batch_preds,axis=-1,direction='DESCENDING',stable=False,name=None)
        
        return tf.reverse(preds_idx,[0])```

In normal mode without micro batched enable, when i passed a dataframe with two rows [ […] , […]], I would get a prediction with 2 rows as well.

Screenshot 2020-11-15 at 1.54.12 PM.png

For extra information of the model

This is the model summary of our model. It is built using keras functional api. We had to specify a batch size because we are using stateful GRU. When loaded as TensorflowSavedModel, the batch size shape will be `None`  .

Let me know if any extra information is needed!

<@U01EKMW9X0A> Hi. It seems that you are padding the batch size to 2048. It’s okay. But the size of returned value is also 2048, which mismatch the input.

<@UQF2L5GQ0> In non batching mode, it will truncate the output according to the length of the input. Are you implying that to have microbatching to work, the output length have to match the input length?

Yeah! When micro-batching enabled, bentoml would pack multiple user requests into one input (the data frame here). Thus, the returned value should have the same batch size with input, makes it possible to split the output accordingly.

I believe bentoml should handle this exception and provide better feedback when user returned mismatched output. I’ll open an issue to track this.

Currently, i believe bentoml will automatically truncate the output according to input length. Anyway I have modify the script according to your suggestion,

```class KerasModelService(bentoml.BentoService):
    @bentoml.api(input=DataframeInput(), mb_max_latency=1000, batch=True)
    def predict(self, df: pd.DataFrame):
        int_oh = np.append(np.zeros((2048 - df.shape[0], df.shape[1])),df,axis=0)
        exp_int_oh = np.expand_dims(int_oh, axis=1)
        batch_preds = self.artifacts.model(exp_int_oh.astype('float32'))
        preds_idx = tf.argsort(batch_preds,axis=-1,direction='DESCENDING',stable=False,name=None)
        rev = tf.reverse(preds_idx,[0])
        return rev[:df.shape[0],:]```
the same error still occurs.

```class KerasModelService(bentoml.BentoService):
    @bentoml.api(input=DataframeInput(), mb_max_latency=1000, batch=True)
    def predict(self, df: pd.DataFrame):
        int_oh = np.append(np.zeros((2048 - df.shape[0], df.shape[1])),df,axis=0)
        exp_int_oh = np.expand_dims(int_oh, axis=1)
        batch_preds = self.artifacts.model(exp_int_oh.astype('float32'))
        preds_idx = tf.argsort(batch_preds,axis=-1,direction='DESCENDING',stable=False,name=None)
        rev = tf.reverse(preds_idx,[0])
        return "testing"```
When i tested with a dummy string above, it will automatically truncate the string “testing” according to the size of the batch input, and it doesn’t throw an error for `i.status` , which makes me think that it is not the output length mismatch that causes this error to occur

After more debugging, i found out that `rev` is type of `EagerTensor` . After converting it to a `ndarray` and return it, microbatching works!

Congratulations. But EagerTensor should work, too. How about explicitly specifying the output adapter `output=TfTensorOutput()` like this?
```
class KerasModelService(bentoml.BentoService):
    @bentoml.api(input=DataframeInput(), output=TfTensorOutput(), mb_max_latency=1000, batch=True)
    def predict(self, df: pd.DataFrame):
        int_oh = np.append(np.zeros((2048 - df.shape[0], df.shape[1])),df,axis=0)
        exp_int_oh = np.expand_dims(int_oh, axis=1)
        batch_preds = self.artifacts.model(exp_int_oh.astype('float32'))
        preds_idx = tf.argsort(batch_preds,axis=-1,direction='DESCENDING',stable=False,name=None)
        rev = tf.reverse(preds_idx,[0])
        return rev[:df.shape[0],:]```


Hi <@UQF2L5GQ0>, I have tested your suggestion and a few more scenario:
1. return Eager Tensor
2. return a Tensor by using `convert_to_tensor`
3. explicit specify output adapter `TfTensorOutput()`
4. creating a dummy tensor `tf.constant([[1.0, 2.0, 3.0], [4.0, 5.0, 6.0]])` 
None of these work, all of it throws the same error as i shown above

I am testing with a single request with the bentoml server running at microbatching mode. Perhaps there might be some incompatibility with tensor outputs?

I guess so. Thank you. Then it is a bug of TfTensorOutput now.

Yeah i supposed! Do you want me to create an issue for this bug?

<@U01EKMW9X0A>
<https://github.com/bentoml/BentoML/pull/1247>
 <https://github.com/bentoml/BentoML/pull/1251>
My bad. Fixed &amp; added to CI.

Thanks <@UQF2L5GQ0>. Do you guys recommend to pull the latest code directly or wait for an official release to have this bug fixed

We recommend to use the official release, a new release is around the corner and we will be back to more frequent release cycle soon.

Hi <@U01F0D8L4HX> - yes you are right it did not make it, I haven’t got a chance to look into it yet so it won’t be available in the upcoming release either

I plan to revisit all our pytorch support work right after the next release, including documentation, torch script and pytorch lightening support

Let me know if you are interested in contributing or would be interested to hear about a workaround before that’s officially supported in the library

I’ll have to learn a bit more about bentoml first I guess, but sure, I could try to help out if I see some achievable task in this regard...

In BentoML, the BentoSerivce `@api`  function is where we recommend putting the preprocessing code, here you can find some example projects: <https://github.com/bentoml/gallery>

You don’t need custom artifact for that

You can load a tf.SavedModel from disk directly as well, check out the docs here:  <https://docs.bentoml.org/en/latest/frameworks.html#tensorflow-2-0>

My model subclasses tf.keras.Model and then overrides the train_step and test_step. I don't think savedmodel serializes these methods yet, so I have to build a new model and load the weights. Doesn't seem like the present artifact works like that, no?

I think this is supported out of the box, I might be wrong - cc <@UQF2L5GQ0>

<@U01EWTTBWVA>
`tf.savedmodel` serializes methods that marked by `tf.function` . If your model is built with Keras API (including subclasses of `tf.keras.Model`) without a  `tf.function`.  `tf.savedmodel` would serializes the default method `__call__` with optional args `training`.

`TensorFlowSavedModelArtifact` is using `tf.savedmodel` to serialize models.

Yep. TensorFlow always gets things complex at beginning:thinking_face:. We just have some new examples using tf.keras API, it may help you: <https://github.com/bentoml/gallery/blob/master/tensorflow/imdb/imdb_text_classification.ipynb>

So would it be okay for me to create a class incorporating the artifacts in the `__init__` of `BentoService`?

Just tried it and it looks like I cannot. Is there any workaround? The reason that I ask is that I already have all my preprocessing packaged in a wrapper class around my tf.keras model. it would be nice to be able to reuse that code

Hey <@U01EWTTBWVA>  - what’s the reason you want to do that?

You can just import that class in your bentoservice API function

what do you mean `create a class incorporating the artifacts in the `__init__` of `BentoService`?`

i have a class with a model instance where the model instance is of type tf.keras.Model

the way my code is designed, this class manages preprocessing and post processing. so it would need to take artifacts.model in its constructor

just use this class in your BentoService class then

you just need to add custom logic in `__init__` right?

As long as you call `super` ’s init method in your overrided init method, it should work

Are these related to your issue?
<https://github.com/bentoml/BentoML/issues/1065>
<https://github.com/bentoml/BentoML/issues/1037>

you can also just lazily initialize this class in your BentoService API callback function

those apply, but lazy initializing works fine for me. that's what I ended up trying

got it, thanks Luke, I will be working on a fix for those two tickets soon, but probably after the next release is out

Hi - are you referring to any example notebook you found online?

You may need to add the `--format`  option

Yes, I am referring to this - <https://notebook.community/kubeflow/examples/prophet_bento_timeseries/bentoml-kubeflow-prophet>

%%writefile uk_data_predict.py

import bentoml
import fbprophet
import pandas as pd
from fbprophet import Prophet
from bentoml import api, env, artifacts, BentoService
from bentoml.adapters import DataframeInput
from bentoml.handlers import DataframeHandler
from bentoml.artifact import PickleArtifact

from bentoml.service.artifacts.common import PickleArtifact

@bentoml.artifacts([PickleArtifact('model')]) #custom pickle artifact

@env(infer_pip_packages=True)

class UKDataPredict (bentoml.BentoService):
    @bentoml.api(input=JsonInput(), batch= True)
    def predict(self, df: pd.DataFrame):
        final_data = df # Figure out the data used for prediction (all rows and all columns)
        predictions = self.artifacts.model.predict(final_data)
        return <http://predictions.as|predictions.as>_data_frame()

I see, that notebook was from 0.8.1, and may have some issue with current version of BentoML

<@U01EJDD02KF> it might be a bug in the JsonInput implementation or the user BentoService code, I don’t think it needs  implementing `--format`

<@U01EJDD02KF> Are you expecting Json data for prediction? Right now, the API endpoint is using `JsonInput` and your predict function is set type to pd.DataFrame.

the API has changed, it should be `api(input=DataframeInput)`

I am expecting to feed a dataframe as input for prediction

@bentoml.api(input=DataframeInput(), batch=True)

*!bentoml run UKDataPredict:20201112213619_0DFFD3 predict --input '{"ds":future_data}'*

Your initial bentoml CLI command is fine

Make sure you setup the correct orient for your request JSON data.  You can find the orient options here: <https://docs.bentoml.org/en/latest/api/adapters.html#dataframeinput>

the way BentoML works is it will attempt to convert your incoming data to the data type you expected.

For your bento service, you set DatframeInput for your API endpoint. BentoML will attempt to convert incoming data to dataframe format and pass that to your predict function.

You can also pass the csv file as input for CLI. Use the option `—input-file` 

Yup...I did that before...but the problem is that I am now getting the result as json which is difficult to comprehend...How do I get the output (prediction) in the dataframe format?

*predictions = self.artifacts.model.predict(final_data)*
*return <http://predictions.to|predictions.to>_data_frame()*

It returns the output in the form of json istelf and

*predictions = self.artifacts.model.predict(final_data)*
*return <http://predictions.as|predictions.as>_data_frame()*
throws an error

BentoML will use your default AWS account that is configured on your system. Make sure configure your AWS account on your system, you can find the link to the instruction in the deployment guide,

<https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-configure.html>

Yup...I have my AWS account set up already. Do I need to install BentoML CLI to run the commands now...If yes, where would I find the Bento CLI?

<https://docs.bentoml.org/en/latest/cli.html>

bentoml cli is installed when you installed bentoml python package using pip

It is throwing an error related to SAM CLI...I have installed it, still getting this error

With Sage Maker, it is also throwing some error...can you please have a look at it?

Hi <@UKH494PHP>  <@UKB4CLKP1>  can you please help me with this

hey Piyush, can you post the error please?

Error: lambda deploy failed: INTERNAL:aws-sam-cli package requires version 0.33.1 Install the package with `pip install -U aws-sam-cli==0.33.1`

When I am trying to install the package using `*pip install -U aws-sam-cli==0.33.1*`, it is throwing an error-&gt;

*ERROR: Could not install packages due to an EnvironmentError: [WinError 5] Access is denied: 'C:\\Users\\Owner\\anaconda3\\Lib\\site-packages\\pywin32_system32\\pywintypes38.dll'*
*Consider using the `--user` option or check the permissions.*

it seems like your issue is related to your env setup. and not related to bentoml. Can you make sure you have the right env setup?

I have installed all the prerequisites but it is throwing an error -&gt;
*ERROR - URPC ERROR ApplyDeployment: [WinError 2] The system cannot find the file specified*
*Error: lambda deploy failed: INTERNAL:[WinError 2] The system cannot find the file specified*

Not sure why I am getting this error...I have run the command -&gt;

*bentoml lambda deploy bento-service-uk-data -b UKDataPredict:20201117184908_40FAB4* 

and I am sure that *UKDataPredict:20201117184908_40FAB4* is present in my local machine

<@U01EJDD02KF> Did u get ur issue resolved?

hey <@U01EJDD02KF> I hope this might help. I'm also required to deploy the service to lambda.
```sudo aws configure set default.region &lt;region&gt;```
Run this and AWS lambda deployment will be successful. Provided you are running this on the machine that has right IAM role attached.

<https://docs.bentoml.org/en/latest/deployment/aws_lambda.html> referring this

Hi Max, sorry there isn’t one, are you familiar with Grafana and Prometheus?

yes, I just haven't looked into how bento does things exactly

can you point me to the code for your integration?

I see, If you start a BentoML server, simply go to the `/metrics`  endpoint

most of the common metrics you would need for a model server is already instrumented there

User can also import prometheus python client and add additional metrics in their BentoService python code, those will also be displayed on the `/metrics`  endpoint

e.g. if I want to do a vanilla AWS deployment, nothing fancy, but I want basic auth to protect my endpoint, what's the way to go?

Got it - there is no built-in support for authentication in BentoML, the most common way to do it is to do it in a frontend proxying layer, for example, using JWT token and verify the token in a Nginx proxy server in front of BentoML api servers

are you thinking about a HTTP basic auth?

nah, just wanted to keep things simple in my question,

np! let me know how it goes, happy to help

<@UKB4CLKP1> follow-up question. if I do e.g. OAuth or whatever with a proxy, how does the swagger UI pick that up? OpenAPI 3.0 has support for many advanced auth solutions, and you can hence log into the Swagger UI for those endpoints. I don't know (yet) how bento builds up the swagger.yml, but I'd assume that could be an issue. Did you ever think about this?

I’m now wondering since i found the extra index url in `requirements.txt` whether those pip install arguments can’t all just be put into `/etc/pip.conf` so that all pip invocations can use it

<@U01BNKWNTMM> I think we probably should also pass the `extra_args` for installing the `bundled_pip_dependencies`

I am not sure to have all of the arguments in `/etc/pip.conf` thou.  Feel like that could cause additional headaches, if the conf is not available on the image build machine, and one extra thing users have to be aware of

thanks for the input <@UKH494PHP>. i created a github issue <https://github.com/bentoml/BentoML/issues/1296>

BentoML has its own async layer built with aiohttp which is optimized for ML model serving 

The flask part is the backend model server, that we could replace with something else later on and does not affect the overall performance that much 

There’s a benchmark project under the bentoml GitHub org page 

yeah I've checked the bentoml/benchmark before, and I'm just thinking maybe a contrast matrix or something like that might make it more attractive. bentoml has made model to service much simpler, I like the idea of packing model with the service code and generating docker image, but some users could hesitate on performance, an intuitive show-out might do some help.

by the way, last few days I'm working on how to make my transformers model run on bentoml with gpu, I solve that problem with a customize bento artifact(found some sample code on slack), but I'm wondering if there is an easier way like changing some setting in bento service?

```# now i do like this
class MyTMA(TransformersModelArtifact):
    def pack(self, model):
        super().pack(model)
        self._model["model"].cuda()



@bentoml.env(infer_pip_packages=True)
@bentoml.artifacts([MyTMA("albert")])
class TransformerService(bentoml.BentoService):
    @bentoml.api(input=JsonInput(), batch=False)
    def predict(self, parsed_json):
        pass```

`optimized for ML model serving`

Just curious, beside batching algorithm, what are the optimization that you guys have made?

<@U01EKMW9X0A> Micro-batching is the major approach. We also made other optimizations like Dataframe merging (calling `pandas.read_json` in a loop is too slow), backpressure control (a common problem for async apps).

Yeah, appling `fastAPI` or `sanic` to the model server (bentoml also has an asyncio batching server in the front) wouldn’t make it faster. Normal web servers are IO-intensive, but inferences of ML models are CPU-intensive.

The exception here is the batching server. It holds thousands of connections from clients, merges inputs to batch, and dispatches to model servers. They are IO-bound. That’s why it's implemented with asyncio. In addition to <https://github.com/bentoml/benchmark>, there is a user in the channel who just made some benchmarks comparing with the tensorflow-serving. Bentoml and tf-serving have similar latency and throughput under high concurrent requests.

<@UQF2L5GQ0> if you are referring to me, yes, I did a benchmark comparing with tf-serving. on low load, few request at a time, tf serving will be able to achieve very low latency. I supposed this is due to tf serving not needing to have python overhead. but at high load thousands RPS, bentoML are able to match the throughput and beat the latency of tf serving. for some reason the response time for bentoML is very stable, while tfserving has higher variation in response time (difference between p5, p50 , p95 response time is huge)

hi <@UQF2L5GQ0>, I compare bert with bentoml and tf-serving. And I get a huge gap of the performance. Betoml will get lots of 429 response and slow responses while tf-serving can success fast and completely. Do you have any ideas how to improve the service?

<@U01BHLCJL81> I also tried BERT with bentoml. There are two possible reasons for your problem:
• Can you find many lines like `triggered tf.function retracing` in the logs? The restracing is very expensive. Here is the solution: <https://github.com/tensorflow/tensorflow/issues/38561#issuecomment-613894395> . `@tf.function(experimental_relax_shapes=True)`
• When running with micro-batching enabled, bentoml requires about five requests to preheat the model and the batching layer. During this the micro-batching will not take effect. So at the beginning, some 429's are normal. The performance will gradually reach the best.

<@UQF2L5GQ0> I test my service and find most of the micro-batch will have size 1; only few times will have size larger one. Do I need to tune  mb_max_latency or mb_max_batch_size ?

Also, when I want to increase our RPS but I will got lots of 429. Do I need to increase the mb_max_latency?

Thanks for reporting this Brendan, to clarify, did you use the *pip_index_url, pip_trusted_host,  or pip_extra_index_url* option in the `@env`  setting?

<@UKH494PHP> I think lambda handles installing pip dependencies differently, could you take a look?

I was thinking my `@env` setting is part of the problem, I was using *auto_pip_depenencies* which led to my module looking something like *my_module==0.2.0* instead of being set to the local path of the module

Yeah. Lambda is handling dependency differently right now. <@U01GQBPQ28Z> could you share with me your current setup?

Your bento service setup. your previous message is very helpful. Is the `my_module` a private package?  Is it  available on any pypi registry?

If you put your dependencies in a pypi package, then you need to distribute it either to the public <http://pypi.org|pypi.org>, or your own private pypi host to make it available in the runtime

Yes `my_module` is my own private package. If I make my own pypi host will I be able to import packages from the public pypi?

you can use `*pip_extra_index_url*` option to specify your own private pypi host

BentoML can not tell if a pypi package installed locally is installed from the public <http://pypi.org|pypi.org> or user’s own private pypi host, that’s the reason for the current behavior.

If your package is small, just put it as a file or files next to your bento_service project, and import them directly - then bentoml will be able to package those local python files automatically

OK I will try these out, thanks for the fast response and the advice!

You can check out the linked repo in this message. <https://bentoml.slack.com/archives/CKC2JP3TM/p1605382781055200|https://bentoml.slack.com/archives/CKC2JP3TM/p1605382781055200>

you can use docker build command directly on a BentoML saved bundle - bypassing the containerize CLI command 

<@UKB4CLKP1> ah, so the saved bundle would have the dockerfile included.

Thanks <@UKH494PHP>, that solution requires YataiService and we were contemplating whether to implement YataiService or not because we already have MLFlow in place for model registries and artifact.

Would like to know if you guys have any plans for integration with MLFlow in the future!

<@U01EKMW9X0A> These utils may help you: <https://github.com/bentoml/BentoML/blob/master/tests/integration/utils.py> . They are not official APIs now of course.

This may help with your workflow, although it is not yet released <https://github.com/bentoml/BentoML/pull/1315>

<@U01GJTBD25Q> I guess you can open a usual pdb session in there, in the end it is just a Flask app.

... or using the werkzeug debugger. You have a full remote console on: `<http://127.0.0.1:5000/console>`

I notice how it will still save an `environment.yml` (conda) file when I pack it as a service though.

it’s not used if you did not set any conda deps 

So there will be no bloat of conda in the final docker image if there are no conda deps, correct?

The default docker base image still have conda installed, but user can choose to use a different docker image without conda

`bloat of conda`  do you mean extra conda dependencies that you don’t need?

Yeah I have bad experience with build times, image size, and misc. other stuff when it comes to conda. However, after testing the latest slim base images the builds went really fast and the image size was very small (below a gig), so quite great :smile:

that’s a great suggestion! We have been discussing this internally, and we think this should be on our roadmap especially considering the requests for adding support for batch inference on apache spark

I would love to have this as it would integrate so well with Argo workflows, I can simply pass the parquet file as an input artifact and work exclusively with yml files specifying the docker image and cmd.

Is google bucket the same thing as google cloud storage?  Google cloud storage is supported as repo-base

Oh! I really struggled to find any documentation on which cloud providers are supported except of `s3`

Sorry about missing documentation and guide on this. 

<https://docs.bentoml.org/en/latest/cli.html?highlight=s3#bentoml-yatai-service-start>

Yeah. The doc string is out dated. I will update this. Thanks for pointing it out

And I'm glad to learn it's supported :smile:

For google cloud storage. Use `gs://` instead of s3

Feel free to ping me if you run into any issue with this. Happy to help

could you try  install grpcio individually and see if it works?
```pip install grpcio&lt;=1.27.2```

<@U01JFK5T7N0> What's your python version? One of bentoml's dependency `py_zipkin` depends on `thriftpy2`
Can you try install py_zipkin, and tell me what happen?

image.png

installed graphico, its python 3.6.8 and when i try to install py_zipkin getting same error previous

Can you try install `thriftpy2` by itself?

Could you also report this issue to thriftpy2?  We will take a look and see if there’s an workaround

<@UKB4CLKP1> pls share thriftpy2 slack channel please ?

Best place is probably their github page?

<https://github.com/Thriftpy/thriftpy2/issues>

<@UKB4CLKP1> FYI raised thriftpy2 issue in github <https://github.com/Thriftpy/thriftpy2/issues/153>

rhel8.2_bentolml_logs.txt

RHEL 8.2 actual Bentolml installation error logs

Is there anyone who know when a new version with <https://github.com/bentoml/BentoML/pull/1352> is going to be released? I am having a trouble using S3-compatible storage for Yatai server :joy:

message (2).txt

Hi <@U01GQBPQ28Z> Thank you for reporting it

I havent encounter this issue before I will do more digging.  It seems like When lambda function is starting, it can't find the s3 bucket that store the extra dependencies.

Can you share your python version, and BentoML version?

PS. We are working on a better Lambda support after AWS introduced container support for Lambda. Stay tuned!

I have tried on different Python versions (3.7/3.8) and BentoML (0.8.3/0.11.0)

"requirement.tar does not exist, downloading from btml-dev-bentomltest-ec9041" is confusing to me because the file in the bucket is requirements.tar but I am not sure if that is just a typo in the error or not

Edit: Merely a typo

Is there anywhere I can stay in the loop for development of Lambda support and learn more about the deployment process? I would love to help in any way I can

<@U01GQBPQ28Z> Definitely!  We have an open issue: <https://github.com/bentoml/BentoML/issues/1289>
That outline what AWS announced and how should we move forward with with container

Also, can you open an issue on this in the github <https://github.com/bentoml/BentoML/issues/new/choose>. Let's track it there and make sure this is resolved

I updated the bug report with my findings
<https://github.com/bentoml/BentoML/issues/1412>

Hi <@U01GQBPQ28Z> this is great find. We will get it fixed. Also. We are working on use container support  for lambda function <https://github.com/bentoml/BentoML/issues/1289|https://github.com/bentoml/BentoML/issues/1289>. It will simplify the process and make lambda deployment less error prone 

Yes, you can only expose one inference API endpoint for now.

Are you looking for expose all inference API endpoints through Sagemaker?

Hi Bo, thanks for the response.
We are testing out each of the different deployment options to gauge which suits our use case the best.
It seems like deploying to Lambda with a docker container will be the best for us, but I believe the functionality is still being built into BentoML.
Is this correct?

Hey Ryan, would you able to describe your use-cases, so we can help you figure out which one fit you better?

Yes, you are correct, we are in the process of implementing container support for Lambda.  Currently Azure Function (similar to Lambda) has container support.  If you are using Azure cloud, this is something I would recommend.

Hi <@UKH494PHP>, thank you again for this great response!

Unfortunately, I cannot speak to the use case in depth because I am new to the team and do not have to the knowledge yet. My role is exploring the model serving options that are available and how they integrate into Azure/AWS etc.

Yes, I see that Azure Function has a painless deployment, we will look into that further!

Furthermore, it really wasn't much work to deploy a containerized saved Beno Service to Lambda as the bento codebase is so well written - thank you!

looking at the code i can see <https://github.com/bentoml/BentoML/blob/506d5026f89916b684493198db5bb2707a11537f/bentoml/yatai/yatai_service.py#L44>

Wondering if it would be possible to set `CUSTOMHEADER: TOKEN_VALUE` or something?

Would this be accepted upstream?

For Sagemaker deployment, BentoML requires docker access to build image, and AWS credentials with permissions for ECR and Sagemaker, and optional permissions such as S3, cloudwatch and others base on your inference service's need.  You can read more about the sagemaker permission here: <https://docs.aws.amazon.com/sagemaker/latest/dg/sagemaker-roles.html#sagemaker-roles-createmodel-perms>