This message was deleted.

One possible solution I see is to use

tag.path

and load the tokenizer first:

tag = bentoml.transformers.get(model_name)
tokenizer = transformers.AutoTokenizer.from_pretrained(tag.path)
pipeline = bentoml.transformers.load_model(tag, trust_remote_code=True, torch_dtype=torch.bfloat16, tokenizer=tokenizer)

Is this the desired way of doing this? I’m suspecting it might have something to do with how the pipeline/config is setup for this falcon model, and thus causing some inconsistency with bentoml backend.

I think to support pipeline tokenizer, you should be able to add the tokenizer into the custom_objects, then pass it to the load_model bentoml.transformers.load_model(…, tokenizer=tag.custom_objects[“tokenizer”])

Got it. Then I think it’s going to be similar to how I resolved this issue. Since the tokenizer is saved together with the pipeline anyways, it can be loaded using the local file path.