Hey all,
First of all, thanks for Promptfoo — I’m loving it!
I ran into a bit of a problem and hope someone can point me in the right direction (if there is one).
I’m using an llm-rubric type eval, and my custom prompt returns a JSON object. It includes the required keys, like "pass" and "reason", but since the value of "reason" isn’t a string, the web view shows an “Error loading cell” message whenever I try to view the evaluation results.
Is there any way to post-process the llm-rubric output? Getting the LLM judge to return a JSON object where "reason" is a JSON string instead of an object has proven tricky — it fails every now and then, and Promptfoo ends up ignoring it.
Thanks in advance!