My rubricPromptis set up like this:

defaultTest:
  options:
    rubricPrompt: |
      You are an expert factuality evaluator. Compare these two answers:

      Question: {% raw %}{{input}}{% endraw %}
      Reference answer: {% raw %}{{ideal}}{% endraw %}
      Submitted answer: {% raw %}{{completion}}{% endraw %}

      Determine if the submitted answer is factually consistent with the reference answer.
      Choose one option:
      A: Submitted answer is a subset of reference (fully consistent)
      B: Submitted answer is a superset of reference (fully consistent)
      C: Submitted answer contains same details as reference
      D: Submitted answer disagrees with reference
      E: Answers differ but differences don't affect factuality

      Respond with JSON: {"category": "LETTER", "reason": "explanation"}

and the eval works as expected. But, the only way I can find to view the JSON response of the grader is by turning

--verbose

on. The 'category' selection, for instance, isn't available in the dashboard or json outputs. I can pipe the command output to a file and jq/grep through that, but I feel like I'm probably missing a better way to grab that info?

@User On the dashboard, when hovering over the output of a specific tes, small symbols appear on the top. By clicking on the magnifying glass symbol ("View output and test details"), you can see the prompt, original output meta data as well as the output of all assertions - in your case the output of the llm-grading. I hope I was able to help. https://cdn.discordapp.com/attachments/1410035995273068728/1411960461804699790/image.png?ex=68b68e27&is=68b53ca7&hm=e6d7d30337b0b2d940c8b169170ca91dee6bd6f22c3dee26c462b063ec44b182&