llm-rubric produced malformed response: {'pass': True, 'score': 4, 'reason': "The submitted answer is highly relevant, addresses the user question comprehensively.
i
ian
01/29/2024, 3:28 PM
out of curiosity, are you overriding the provider for the
llm-rubric
assertion? You may also want to try
model-graded-closedqa
and see if that works better with your test cases
p
piyush_27224_71903
01/30/2024, 8:56 AM
Yes we are overriding the provider for llm-rubric assertion. Yes we already have model-graded-closedqa and that also works intermittent fine, i mean sometimes good sometimes even with all assertions ok and mark it as 'N'.
i
ian
01/30/2024, 3:17 PM
which model are you using to override? I find that anything other than gpt-4 doesn't work very well.
ian
01/30/2024, 3:18 PM
This PR improves the reliability of
llm-rubric
by forcing a JSON-formatted response for the default provider. It will be in the next release: