https://promptfoo.dev/ logo
Test for my API endpoint
# ❓|questions
u
hi, I have an API that receives a prompt string and return a json with the text response plus another field that return an evaluation of the prompt with "bad" prompt or "not bad", how can I use promptfoo to test this API and have an evaluation on how correct are the evaluations?
i
check out the HTTP provider https://www.promptfoo.dev/docs/providers/http/ you can just transformResponse to extract good/bad alternatively, you could have more control by writing your own custom provider in [python](https://www.promptfoo.dev/docs/providers/python/) or [javascript](https://www.promptfoo.dev/docs/providers/custom-api/)
u
that is fine, I can use http raw request and then use transformResponse to parse the response, but how exactly promptfoo can do evaluation once I have a response? just based on the text response from the model or it can do something with the already scored strings "bad" or "good" in the response (the API returns a score for prompt and response string)?