https://bentoml.com logo
Join Slack
Powered by
This message was deleted.
# ask-for-help
s
This message was deleted.
c
Hi Boris, thanks for reporting this issue, we will look into it!
@Jian Shen Yap could you help take a look?
b
Thanks! That’s the last thing that’s preventing me from start using in production
j
Hey @Boris Nadion I'll take a look and let you know soon!
b
Hey @Jian Shen Yap, don’t want to rush you, but do you have some timeline in mind, I want to plan the efforts accordingly, thanks in advance!
j
Hey! @Boris Nadion the fix should come by today!
b
j
yeap! it's probably a misconfiguration in the 
bentofile.yaml
, i'm running some test to make sure it works
b
Thanks a lot!
j
@Boris Nadion are you using the 
pip
version or from the source?
b
Copy code
❯ pip --version
pip 23.2.1 from /Users/bnadion/.pyenv/versions/3.11.4/lib/python3.11/site-packages/pip (python 3.11)
is that what you mean?
ahh, I use the pip version, I can install from GitHub
j
sorry I should have been clearer, are you using the version of 
clip-api-service
installed from pip or from the github source
got it!
b
so, should I just install it from the source now?
j
I'll give you an update by EOD
b
Thanks!
j
Hey @Boris Nadion, thanks for your patience. I have made the nessecary fixes to two of the issues that you've raised. We will release a 
0.1.11
version tonight or tomorrow. if you wanted to test it out, you can either build it from source or use the wheel here https://github.com/bentoml/CLIP-API-service/releases/tag/v0.1.1
thanks for your feedback and spotting the errors!
b
Thanks! I will do that shortly.
@Jian Shen Yap smth is still not right, when I run
Copy code
clip-api-service serve --model-name=ViT-B-32:laion2b_s34b_b79k
I get a correct 512 dim array response when I build
Copy code
clip-api-service build --model-name=ViT-B-32:laion2b_s34b_b79k
then run locally
Copy code
bentoml serve clip-api-service:432ybmdhnccxcar6
or deploy (push) I get a wrong 768 dim array response, something is still not right Thanks!
It looks like openai-clip/vit-large-patch14 is still downloaded and used even if I remove it from the models folder
j
did you install it with the new wheel?
b
yes, passing ENV var appears to be not enough, get_clip_module should be responsible to bring correct yaml with correct model config https://github.com/bentoml/CLIP-API-service/blob/bdcb56f6920e662aee55b700860086ef38bd7a0f/src/clip_api_service/build.py#L14
i will try to debug that
j
what is the output of your 
bentoml get <tag>
b
Copy code
service: clip_api_service.service:svc
name: clip-api-service
version: 432ybmdhnccxcar6
bentoml_version: 1.1.6
creation_time: '2023-10-10T12:31:31.386566+00:00'
labels: {}
models:
- tag: openclip:vit-b-32.laion2b_s34b_b79k
  module: bentoml.pytorch
  creation_time: '2023-10-08T06:29:39.416708+00:00'
runners:
- name: clip_model_runner
  runnable_type: OpenClipRunnable
  embedded: false
  models:
  - openclip:vit-b-32.laion2b_s34b_b79k
  resource_config: null
apis:
- name: encode
  input_type: JSON
  output_type: JSON
- name: rank
  input_type: JSON
  output_type: JSON
docker:
  distro: debian
  python_version: '3.11'
  cuda_version: null
  env: null
  system_packages: null
  setup_script: null
  base_image: null
  dockerfile_template: null
python:
  requirements_txt: null
  packages:
  - bentoml
  - transformers
  - accelerate
  - optimum
  - pydantic
  - Pillow
  lock_packages: true
  index_url: null
  no_index: null
  trusted_host: null
  find_links: null
  extra_index_url: null
  pip_args: null
  wheels: null
conda:
  environment_yml: null
  channels: null
  dependencies: null
  pip: null
correct model name
I tried to rebuild and then 
bentoml serve clip-api-service:d7yav6thow54car6
it still shows wrong number of dimensions which means it’s built with wrong model, or being served with a wrong model, because I see both models in the 
models
folder
probably the problem is with serve, if I remove the model from the folder that’s the output:
Copy code
bentoml serve clip-api-service:d7yav6thow54car6
`text_config_dict` is provided which will be used to initialize `CLIPTextConfig`. The value `text_config["id2label"]` will be overriden.
`text_config_dict` is provided which will be used to initialize `CLIPTextConfig`. The value `text_config["bos_token_id"]` will be overriden.
`text_config_dict` is provided which will be used to initialize `CLIPTextConfig`. The value `text_config["eos_token_id"]` will be overriden.
2023-10-10T17:00:52+0300 [INFO] [cli] Using the default model signature for Transformers ({'__call__': ModelSignature(batchable=False, batch_dim=(0, 0), input_spec=None, output_spec=None), 'forward': ModelSignature(batchable=False, batch_dim=(0, 0), input_spec=None, output_spec=None), 'generate': ModelSignature(batchable=False, batch_dim=(0, 0), input_spec=None, output_spec=None), 'contrastive_search': ModelSignature(batchable=False, batch_dim=(0, 0), input_spec=None, output_spec=None), 'greedy_search': ModelSignature(batchable=False, batch_dim=(0, 0), input_spec=None, output_spec=None), 'sample': ModelSignature(batchable=False, batch_dim=(0, 0), input_spec=None, output_spec=None), 'beam_search': ModelSignature(batchable=False, batch_dim=(0, 0), input_spec=None, output_spec=None), 'beam_sample': ModelSignature(batchable=False, batch_dim=(0, 0), input_spec=None, output_spec=None), 'group_beam_search': ModelSignature(batchable=False, batch_dim=(0, 0), input_spec=None, output_spec=None), 'constrained_beam_search': ModelSignature(batchable=False, batch_dim=(0, 0), input_spec=None, output_spec=None)}) for model "openai-clip:vit-large-patch14".
2023-10-10T17:00:58+0300 [INFO] [cli] Environ for worker 0: set CPU thread count to 10
2023-10-10T17:00:58+0300 [INFO] [cli] Prometheus metrics for HTTP BentoServer from "clip-api-service:d7yav6thow54car6" can be accessed at <http://localhost:3000/metrics>.
2023-10-10T17:00:59+0300 [INFO] [cli] Starting production HTTP BentoServer from "clip-api-service:d7yav6thow54car6" listening on <http://0.0.0.0:3000> (Press CTRL+C to quit)
Using the default model signature for Transformers
and then wrong model is downloaded and served
that’s the call stack from `bentoml serve xxx`:
Copy code
File "/Users/bnadion/.pyenv/versions/3.11.4/lib/python3.11/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1204, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1176, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1147, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 690, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 940, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "/Users/bnadion/bentoml/bentos/clip-api-service/d7yav6thow54car6/src/clip_api_service/service.py", line 49, in <module>
    logit_scale = np.exp(bento_model.info.metadata.get("logit_scale", 4.60517))
  File "/Users/bnadion/bentoml/bentos/clip-api-service/d7yav6thow54car6/src/clip_api_service/runners.py", line 14, in get_clip_runner
    runner = bentoml.Runner(
as you can see in init_model doesn’t get default param from yaml, it takes it from env var or uses default, env var doesn’t set here because we just run it as is
the temporary hack is to change DEFAULT_MODEL_NAME inside the bento which was just build before pushing it to cloud
j
thanks for spotting!
will make a release soon!
b
…also did not help because in the cloud the deployment failed:
Copy code
2023-10-10 18:08:07.319	
ModuleNotFoundError: No module named 'open_clip'
2023-10-10 18:08:07.319	
2023-10-10T15:08:07+0000 [ERROR] [runner:clip_model_runner:1] Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/starlette/routing.py", line 705, in lifespan
    async with self.lifespan_context(app) as maybe_state:
  File "/usr/local/lib/python3.11/contextlib.py", line 204, in __aenter__
    return await anext(self.gen)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/bentoml/_internal/server/base_app.py", line 75, in lifespan
    on_startup()
  File "/usr/local/lib/python3.11/site-packages/bentoml/_internal/runner/runner.py", line 317, in init_local
    raise e
  File "/usr/local/lib/python3.11/site-packages/bentoml/_internal/runner/runner.py", line 307, in init_local
    self._set_handle(LocalRunnerRef)
  File "/usr/local/lib/python3.11/site-packages/bentoml/_internal/runner/runner.py", line 150, in _set_handle
    runner_handle = handle_class(self, *args, **kwargs)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/bentoml/_internal/runner/runner_handle/local.py", line 27, in __init__
    self._runnable = runner.runnable_class(**runner.runnable_init_params)  # type: ignore
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/bentoml/bento/src/clip_api_service/models/openclip.py", line 160, in __init__
    self.model = bento_model.load_model().to(self.device).eval()
                 ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/bentoml/_internal/models/model.py", line 379, in load_model
    self._model = self.info.imported_module.load_model(self, *args, **kwargs)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/bentoml/_internal/frameworks/pytorch.py", line 79, in load_model
    model: "torch.nn.Module" = torch.load(file, map_location=device_id)
Please let me know if I can help anyhow
j
Your findings are already helpful! I'm currently working on a fix as well as testing it out end to end with bentocloud! I'll keep you posted by the EOD
b
Thanks!
Hey, any news?
j
Hey! Just pushed a fix, and currently testing on the cloud
it's a workaround where I inject the model name into the service code with text templating
If you are planning to use this in production, I would love to do a refactoring to make it more user friendly like our other repository OpenLLM and OneDiffusion.
Both OpenLLM and OneDiffusion uses a generated 
service.py
file
Hey I have released 
v0.1.2
in pypi, please give it a try!
b
Thanks, I will check that shortly! I do plan to use that in production.
👍 1
j
Sure. We could work together to make the tool more friendly and robust
And thank you for your feedback!
b
Everything works perfect, thanks!
❤️ 2
Open in Slack
PreviousNext
Powered by