https://promptfoo.dev/ logo
Custom calls to vLLM/TGI and benchmark them
# ❓|questions
e
Hi there, i've seen that there is support for calling to provider vLLM https://www.promptfoo.dev/docs/providers/vllm . Awesome! So if I host my own stuff on vLLM I could use promptfoo for that - right? How about TGI? I didn't see official support but probably this provider https://www.promptfoo.dev/docs/providers/webhook will do? Second part of my question: I want to see in detail what metrics my prompt causes on the machine - so metrics from TGI/vLLM. Would that be possible? I love the features of promptfoo that I see in the docs. But in terms of "performance benchmark" I've only discovered the average run time + used tokens. Is there more to enhance? If yes, how?
i
You're right that there isn't a TGI provider, but definitely something that would be worth adding promptfoo. In the meantime, the best way to use a TGI endpoint is probably by implementing a custom [python](https://promptfoo.dev/docs/providers/python/) or [javascript](https://promptfoo.dev/docs/providers/custom-api) provider