https://promptfoo.dev/ logo
Comparing Chatbot actual response Output with Cach...
# ❓|questions
p
I have been using promptfoo for various evaluations. However I’m currently exploring the performance of a chatbot that’s been integrated with GPT-4, focusing on its nondeterministic output. My goal is to evaluate the chatbot’s responses using promptfoo assertions for a more in-depth analysis. Is there a method to directly compare the chatbot’s actual output with a set of expected responses I’ve cached, bypassing the intermediary provider (running the eval against provider)? My requirement is to compare the chatbot output response to a user input. Any guidance or tips on how to accomplish this would be greatly appreciated. Thanks in advance!
i
Depending on what language your chatbot is written in, I would use a [python](https://promptfoo.dev/docs/providers/python), [javascript](https://promptfoo.dev/docs/providers/custom-api), or [custom](https://promptfoo.dev/docs/providers/custom-script) provider. Using these providers, you can fetch the nondeterministic chatbot response directly using your chatbot logic. Then, I would set up [assertions](https://promptfoo.dev/docs/configuration/expected-outputs/deterministic) to test for your expected responses.
Let me know if that makes sense. If your assertion logic is complicated, you can use a custom [javascript](https://promptfoo.dev/docs/configuration/expected-outputs/javascript) or [python](https://promptfoo.dev/docs/configuration/expected-outputs/python) assertion.
p
Thank you for your prompt reply, @ian Promptfoo has significantly helped our initial LLM evaluations, and your support in maintaining such a versatile framework is greatly appreciated. To give you a better idea of what I’m working on: we have a webpage featuring a chatbot that allows our customers to inquire about various topics. I’m currently developing end-to-end UI testing using Playwright to simulate interactions between the customer and the chatbot. After each customer input, the chatbot generates a response. Utilizing Playwright, I can capture this response and store it in a variable. The next step involves evaluating this output using promptfoo to ensure it meets our expected standards. Could you advise on the most efficient method to accomplish this? Specifically, would employing a JavaScript provider be the best approach in this scenario?
i
Yeah, in general I think the javascript provider is the way to go! You can approach this in a couple different ways: 1. Have the playwright tests run separately and generate YAML/JSON prompt configs. Then run those configs directly in promptfoo. 2. Run the playwright tests separately, and write the outputs to a file. Then have a thin Javascript provider read this file and return the contents 3. Have the Javascript provider also run the playwright script, simulate the interactions, and return the results.
#3 is probably the most straightforward if you plan to always run evals on the e2e ui testing.
p
Appreciate the insight, @ian On the matter of #1 regarding YAML/JSON prompt configurations: I’ve noticed that all examples typically involve using a provider or model, such as GPT or another LLM. In my situation, as I already possess the output, how can I avoid or bypass executing the promptfoo evaluation through an LLM model? And just leverage local data I have.
i
You could use the "standalone assertions" feature. Here's an example: https://github.com/promptfoo/promptfoo/pull/368
This lets you package your predefined outputs into
outputs.json
and run against a list of assertions directly
p
Never mind, found it. This example solved it for me https://www.promptfoo.dev/docs/providers/custom-api/
i
That works too! The standalone assertions feature is basically a custom API provider that just returns the output
p
message has been deleted
You are awesome 👏
i
np! let me know if you have any more questions
p
@ian I experimented with using standalone assertions for evaluating outputs, but encountered a limitation when working with multiple outputs. Despite having three distinct outputs in outputs.json and matching assertions in assert.yaml, it seems there isn’t a direct one-to-one correlation. This discrepancy became apparent when referencing the ‘hello’ example you shared, which included three different assertions. However, only the first assertion, ‘icontains hello,’ was recognized and applied. Is this behavior indicative of a bug, or is there a detail I’m overlooking?
What are you seeing?
Looking at the image in this PR, it’s the same response I got https://github.com/promptfoo/promptfoo/pull/368
For all 3 output eval it’s expecting output to contain hello.
i
That makes sense to me - all 3 asserts in asserts.yaml are applied to all outputs. In the example case provided, two of the outputs fail the
icontains
assert
So it's working as intended. Did you have something different in mind?
p
Thanks @ian . I am good with it. Your suggestion helped me and I have implemented e2e using playwright and promptfoo. The standalone assertion was the solution we decided to go with. Appreciate your insights and keeping the promptfoo framework updated
2 Views