https://promptfoo.dev/ logo
asserting on JSON output
# ❓|questions
n
Running promptfoo v0.43.1 Using azure openai provider in JSON mode. We are using javascript assertions to inspect the generated JSON response. Based on the documentation (https://www.promptfoo.dev/docs/guides/evaluate-json#ensuring-the-validity-of-specific-json-fields) I assumed that
output
was already parsed as JSON. Which would allow us to write something like: > output.analysis[0].tag.includes('Fun') However, the
typeof output
is string. So for each javascript assertion we need to perform
JSON.parse(outout)
before interacting with the output as JSON. Using
JSON.parse(output)
works like a champ ... but the docs seem to imply that JSON.parse isn't necessary. Am i doing something wrong?
taking a closer look at the example in the documation section i linked to above i notice something else really odd: > # Parse the JSON and test the contents > - type: javascript > value: output.color === 'yellow' && ["Guatemala", "Costa Rica"].includes(outputObj.location) The snippet references both
output
and
outputObj
. i tried
outputObj
thinking maybe that is the JSON parsed version of the output. No dice ... the output obj does not exist.
i
hmm, I'll take a look. Thanks for flagging
n
@ian better than 50/50 chance that this is a "me" problem. let me know what you find ... and a preemptive sorry if i just missed something in the documentation.
i
This is on me. Somehow that doc fell out of date - sorry about that 😦 I've updated the doc you linked and also created a json-output example here after verifying that everything is working locally: Let me know if you run into any issues with this!
n
thanks @ian - you rock. i started using the transform when i reread the JSON guide in the docs a few dozen times 🙂 Makes like a lot easier when when you have a slew of JSON related assertions under a single assert property in a test case.
3 Views