https://promptfoo.dev/ logo
Post-processing llm-rubric response
# questions
a
Hey all, First of all, thanks for Promptfoo — I’m loving it! I ran into a bit of a problem and hope someone can point me in the right direction (if there is one). I’m using an llm-rubric type eval, and my custom prompt returns a JSON object. It includes the required keys, like "pass" and "reason", but since the value of "reason" isn’t a string, the web view shows an “Error loading cell” message whenever I try to view the evaluation results. Is there any way to post-process the llm-rubric output? Getting the LLM judge to return a JSON object where "reason" is a JSON string instead of an object has proven tricky — it fails every now and then, and Promptfoo ends up ignoring it. Thanks in advance!
u
Hey, I can help look into this. Can you share an example promptfooconfig.yaml that reproduces this error?
a
Hi, thanks in advance! I attached the prompt I'm using. I checked the detailed back-and-forth between Promptfoo and the API, and the JSON structure comes through fine. It just can't be displayed with
promptfoo view
. https://cdn.discordapp.com/attachments/1427239218463113236/1427566624864600075/prompt.md?ex=68ef5486&is=68ee0306&hm=21150180e94117ba44d3c44058e8ba4c499ec4188c73bbb89cd3302d15f6c0a5&