https://promptfoo.dev/ logo
transformResponse json field evaluation
# ❓|questions
u
I have an REST API endpoint the receive a prompt and return a json with the model response plus a sort of score for the input prompt like { "response": "response text", "score":"bad or good" } , now I have the json response in transformResponse, how can I make promptfoo the evaluation based on the field score in the json? I mean instead of trying to interpret the raw text in the response, just parse the json and check for the field "score", does it make sense, and it is possible?
m
The provider response can be any shape (string, json, object, etc.). You can remove the transformResponse and pass through the entire JSON or a subset of fields. What assertions do you want to run over your response? If for example you want to validate that the score is above a threshold you could put for example
Copy code
- type: javascript
   value: output.score > .8 # or JSON.parse(output).score > .8
u
ah cool, the javascript type also supports loops? for example if the json contains multiple scores and I want to loop all of them and check for a threshold?
m
javascript supports anything you can do in the language, including things like calling another LLM. I recommend defining them externally if you need more than a couple lines. See https://www.promptfoo.dev/docs/configuration/expected-outputs/javascript/#external-script