https://promptfoo.dev/ logo
Getting llm-rubric produced malformed response ver...
# ❓|questions
p
llm-rubric produced malformed response: {'pass': True, 'score': 4, 'reason': "The submitted answer is highly relevant, addresses the user question comprehensively.
i
out of curiosity, are you overriding the provider for the
llm-rubric
assertion? You may also want to try
model-graded-closedqa
and see if that works better with your test cases
p
Yes we are overriding the provider for llm-rubric assertion. Yes we already have model-graded-closedqa and that also works intermittent fine, i mean sometimes good sometimes even with all assertions ok and mark it as 'N'.
i
which model are you using to override? I find that anything other than gpt-4 doesn't work very well.
This PR improves the reliability of
llm-rubric
by forcing a JSON-formatted response for the default provider. It will be in the next release: