https://promptfoo.dev/ logo
JSON reply assert property boolean value.
# ❓|questions
d
Hi, I am trying to assert the response returned from the LLM. The LLM is supposed to return a JSON with a number of flag set to true or false. I would like to make sure that these flags are set up correctly. Here is what I have got so far:
Copy code
tests:
  - vars:
      query: 'What are the number of visits?'
    assert:
      # - type: is-json
      - type: contains-json
        value:
          required:
            - read_metric_value
          properties:
              read_metric_value:
                type: boolean
                # value: true  # This doesn't work
Basically I want to check that the response matches something like this:
Copy code
{
  "read_metric_value": true
}
m
Promptfoo supports JSON schemas as the value for our contains-json and is-json assertions. Here's an example of how to set up your schema:
Copy code
providers:
  - echo

prompts: '{ "read_metric_value": true }'

tests:
  - vars:
      query: 'What are the number of visits?'
    assert:
      - type: contains-json
        value:
          type: object
          required:
            - read_metric_value
          properties:
            read_metric_value:
              type: boolean
              enum: [true]
If you change either true to false it will fail. Check out https://json-schema.org/learn/getting-started-step-by-step for a quick tour of JSON schemas (or ask an LLM - I asked Claude to correct your schema because I didn't know the best answer). Hope this helps!
d
amazing thank you!!
2 Views