BentoML

image.png

Hey, what version of openllm are you using?

Let’s send this to the discord as well?

it could be because I had not provided `adapter_map` to `openllm.Runner()` method as I did not know what I should provide `OPENLLM_ADAPTER_MAP` env var

could you please let me know what is `OPENLLM_ADAPTER_MAP` for `google/flat-t5-large` model?

you don’t have to provide the ADAPTER_MAP envvar. the ADAPTER_MAP is only used with custom LoRA layer

Kindly ask to also post this in the discord for tracking

Any chances that you are using quantization?

THis might happen when the model is loaded with quant 8

This is the values passed to `openllm.Runner()`
```model = os.environ.get("OPENLLM_MODEL", "flan_t5")
model_id = os.environ.get("OPENLLM_MODEL_ID", "google/flan-t5-large")
adapter_map = os.environ.get("OPENLLM_ADAPTER_MAP", """{__model_adapter_map__}""")  # openllm: model adapter map

llm_config = openllm.AutoConfig.for_model(model)

runner = openllm.Runner(
    model,
    model_id=model_id,
    llm_config=llm_config,
    bettertransformer=llm_config["env"]["bettertransformer_value"],
    quantize=llm_config["env"]["quantize_value"],
    adapter_map=orjson.loads(adapter_map),
)```
and
```llm_config["env"]["quantize_value"] is (None,)```

What is the configuration you are using for yatai on this model?

aaah, my bad, I set low resources, its working after increasing. thanks <@U01BMCJ29LN>