https://promptfoo.dev/ logo
Run asserts on part of JSON output
# ❓|questions
v
Heya! Is there a way to specify which part of the JSON output you want to run an assert on for a specific test case? For instance, our prompt outputs a JSON object of the following output:
{"field-a": "a", "field-b": "this is the value for field b"}
Is there a way to write the test config similar to the following?
Copy code
yaml
- vars:
    var1: xyz
  assert:
    - type: is-json
    - type: equals
      var: output["field-a"]
      value: "a:"
    - type: similar
      var: output["field-b"]
      value: value for field b
I could have separate test configs/providers that output just the different parts of the JSON output, but it can get messy pretty quick (also thanks in advance, the multiple
__expected
columns have worked great )
i
The way I would do this is with a JS assertion:
Copy code
yaml
assert:
  - type: is-json
  - type: javascript
    value: output.fieldA === 'foo'
unfortunately that will just work for logical assertions, and you wouldn't be able to use things like similarity for
similar
, you could use the
postprocess
directive to do something like this.
postprocess
applies to the entire test case:
Copy code
yaml
tests:
  - postprocess: output.fieldA   # returns only fieldA from the output
    assert:
      - type: similar
        value: foo
Hope this helps!
v
Hey! Thanks for the input. I tried the post processing for the entire test case that you mentioned above, but the assert is still running against the entire output. I am using a python provider if that changes anything. Thanks in advance!