https://bentoml.com logo
Title
n

Nuwan

05/25/2023, 7:44 AM
Hi, I’m trying to save a transformer model and tokenizer. But I am getting error. I wanted to use T5, but its similar for all. I cannot save tokenizer.
import bentoml
from transformers import AutoModel, AutoTokenizer

sum_model = AutoModel.from_pretrained("bert-base-uncased")
tokenizer = AutoTokenizer.from_pretrained("bert-base-uncased")

bento_model = bentoml.transformers.save_model(
    "email_sum_model",
    sum_model,
)
bento_tokenizer = bentoml.transformers.save_model(
    "email_sum_tokenizer",
    tokenizer,

)
Traceback (most recent call last):
  File "/mnt/ssd/mamba/envs/sum/bin/bentoml", line 8, in <module>
    sys.exit(cli())
  File "/mnt/ssd/mamba/envs/sum/lib/python3.10/site-packages/click/core.py", line 1130, in __call__
    return self.main(*args, **kwargs)
  File "/mnt/ssd/mamba/envs/sum/lib/python3.10/site-packages/click/core.py", line 1055, in main
    rv = self.invoke(ctx)
  File "/mnt/ssd/mamba/envs/sum/lib/python3.10/site-packages/click/core.py", line 1657, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/mnt/ssd/mamba/envs/sum/lib/python3.10/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/mnt/ssd/mamba/envs/sum/lib/python3.10/site-packages/click/core.py", line 760, in invoke
    return __callback(*args, **kwargs)
  File "/mnt/ssd/mamba/envs/sum/lib/python3.10/site-packages/bentoml_cli/utils.py", line 334, in wrapper
    return func(*args, **kwargs)
  File "/mnt/ssd/mamba/envs/sum/lib/python3.10/site-packages/bentoml_cli/utils.py", line 305, in wrapper
    return_value = func(*args, **kwargs)
  File "/mnt/ssd/mamba/envs/sum/lib/python3.10/site-packages/bentoml_cli/utils.py", line 262, in wrapper
    return func(*args, **kwargs)
  File "/mnt/ssd/mamba/envs/sum/lib/python3.10/site-packages/simple_di/__init__.py", line 139, in _
    return func(*_inject_args(bind.args), **_inject_kwargs(bind.kwargs))
  File "/mnt/ssd/mamba/envs/sum/lib/python3.10/site-packages/bentoml_cli/bentos.py", line 323, in build
    bento = Bento.create(
  File "/mnt/ssd/mamba/envs/sum/lib/python3.10/site-packages/simple_di/__init__.py", line 139, in _
    return func(*_inject_args(bind.args), **_inject_kwargs(bind.kwargs))
  File "/mnt/ssd/mamba/envs/sum/lib/python3.10/site-packages/bentoml/_internal/bento/bento.py", line 182, in create
    svc = import_service(
  File "/mnt/ssd/mamba/envs/sum/lib/python3.10/site-packages/simple_di/__init__.py", line 139, in _
    return func(*_inject_args(bind.args), **_inject_kwargs(bind.kwargs))
  File "/mnt/ssd/mamba/envs/sum/lib/python3.10/site-packages/bentoml/_internal/service/loader.py", line 137, in import_service
    module = importlib.import_module(module_name, package=working_dir)
  File "/mnt/ssd/mamba/envs/sum/lib/python3.10/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 883, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "/mnt/ssd/projects/AIESA/bento_deploy/service.py", line 30, in <module>
    bento_tokenizer = bentoml.transformers.save_model(
  File "/mnt/ssd/mamba/envs/sum/lib/python3.10/site-packages/bentoml/_internal/frameworks/transformers.py", line 867, in save_model
    "_framework": pretrained.framework,
AttributeError: 'BertTokenizerFast' object has no attribute 'framework'
I’m pretty new to bentoml. Am I doing anything wrong or what is the cause. Thanks in advance.
👍 1
a

Aaron Pham

05/25/2023, 8:00 AM
oh sorry, this is def a bug. Let me put up a PR for this. Thanks for spotting this.
For now what you can do is to save the tokenizer as a custom object
bentoml.transformers.save_model("model", model, custom_objects={"tokenizer": tokenizer})
And you can access this tokenizer like so
bentomodel = bentoml.models.get("model")

tokenizer = bentomodel.custom_objects['tokenizer']
n

Nuwan

05/25/2023, 8:02 AM
Thank you.. I will it. 👍
a

Aaron Pham

05/25/2023, 8:07 AM
I notice that you are also running save inside service.py. This is usually discouraged because
service.py
will be forked to child process when running
bentoml serve
, and you probably don’t want to save the model everytime you serve the service.
Usually save should be done after training, but for the case of transformers, you can create a separate file to run before the actual serving time
n

Nuwan

05/25/2023, 8:08 AM
Oh, Thank you for information. I will use separate file for it.