https://promptfoo.dev/ logo
Model grader produced a malformed response
# ❓|questions
p
I'm running a series of 'model-graded-closedqa' assertion tests on a prompt. Generally it seems to work pretty decently, however I often get something along these lines back: Model grader produced a malformed response: The criterion requires the submission to mention something similar to "X". In the submission, it is stated that "X" This statement clearly indicates Y, which aligns with the criterion. Therefore, the submission meets the criterion. The response is saying "the submission meets the criterion", but it ends up failing the assertion. Any idea?
i
can you try swapping out
model-graded-closedqa
with
llm-rubric
? I've found that llm-rubric is a bit less flaky
model-graded-closedqa uses the exact prompt that OpenAI uses for their evals. So it has that "blessing", but sometimes I find that it does not perform...
p
Seems to be working a lot better!
Will keep testing!
i
cool, let me know if you find that it doesn't work as well
8 Views