https://promptfoo.dev/ logo
Metrics without assertion
# ❓|questions
v
Hi... I am very new to promptfoo, so if the question sounds stupid please be nice. I want to use promptfoo to evaluate an internal fine tuned model. We have a custom evaluation strategy that assigns 2 scores to every response from the model under test. The scores can be individually aggregated across all test cases. I want to integrate this custom eval with prompt foo. I don't want to assert whether something will pass or fail. Is promptfoo the right tool for this?
i
not a stupid question! you can do this with a [custom javascript](https://www.promptfoo.dev/docs/configuration/expected-outputs/javascript/) or [custom python](https://www.promptfoo.dev/docs/configuration/expected-outputs/python/) assertion. you can script the custom evaluation strategy to talk with your internal fine tuned model, does it support an openai-like api? what is serving inference?
v
Currently the inferencing is using ollama. Will move it to vllm at that point. It is fair to assume it will have a openai compatible api. But even this python forces me to assert or am i misreading it?