https://promptfoo.dev/ logo
assert across multiple outputs
# ❓|questions
h
My use case is that I'm hitting a model that I'm hosting and along with testing for quality i want to test whether sampling params like temperature, top_p etc are working as intended. To do that I'd need to analyze a number of outputs from the same input and see how different they are. Wondering if there's a clean way for me to test a set of outputs?
i
best approach here would be to set each version with different sampling params as a provider. Then you can set assertions on each one and compare performance the only assertion that literally compares across multiple providers is model-graded
select-best
h
Got it so each provider would have a different temp etc and then you'd just see how they fare at the various assertions. Really appreciate your work with this project! It's quite deceptively deep, almost feels like it'd benefit from a 101 video breaking down idiomatic promptfoo 😄 Do you forsee future support of a kind of assertion that runs a test across N outputs?