https://promptfoo.dev/ logo
a
While running the promptfoo use case Mistral vs Llama 2: benchmark on your own data on Kaggle with the grader other than OpenAI getting errors. Command: !npx promptfoo@latest eval --grader replicate:meta/llama-2-7b-chat:52551facc68363358effaacb0a52b5351843e2d3bf14f58aff5f0b82d756078c Error: [FAIL] Output is not valid JSON: Sure! I'd be happy to help you with your task. Please provide me with the input and rubric you would like me to evaluate, and I will return the result as a JSON object with the format you specified. Can someone please help how to use the grader effectively for assertions and which open source grader is recommended.
i
Unfortunately you're getting this because Llama doesn't follow the grading instructions. I've tried to experiment with
model-graded-closedqa
(which has a more lenient output format) and Llama-70B, but no luck there either. I haven't found an open-source model that reliably outputs grading in a structured format. It probably needs to be fine-tuned.
In case it helps, here is the
model-graded-closedqa
prompt. It is written by OpenAI: The prompt can be customized by overriding the
options.rubricPrompt
on the test object: It's quite possible there's a format that is Llama-friendly. It's just that no one has written it yet.
a
Thanks ian!! As you mentioned, It seems to be prompt format.