https://promptfoo.dev/ logo
Running multiple assertions on same LLM response?
# ❓|questions
a
Hello, My team has a situation in which we would like to run the results of the same provider against multiple assertions. We want to do this to save on LLM calls and also to maintain ease of interpretation in our results. For a simplified example (see attached), we would want test1 and test2 to be asserting on the same model response. This is also complicated somewhat by our use of a custom provider that wraps our model. Essentially, our model is an LLM chain, and we want the assertions on the different steps to be run on different parts of the same LLM call. We originally had the model return the output of the entire chain as JSON, but it made it hard to read in the output and we would have had to write custom assertions in order to be able to check different steps in the chain. So we made a variable called "step" that tells the model wrapper which part of the chain to return, which made it much more readable. However, I am worried that even if there is a way to make multiple assertions on the same output, in this case the "step" variable would change the "output". My only idea for dealing with this was to make a table in a local database that temporarily "caches" our model's responses in the wrapper based off of the set of variables it was called with, and returns the same response if all the variables except "step" are ones it has used that combo of in the eval run before. I am hoping we don't have to resort to this and am thus coming to y'all for advice. So to summarize, my questions are: 1. Is there a way to specify running different assertions on the same output when the variables were the same? 2. Would it work for our use case with a chain and a "step" variable, or do you have any ideas of how to adjust our usage? 3. If this isn't currently possible, is there anything in the works that would facilitate this use case? Any guidance or suggestions from the community would be greatly appreciated.
i
Hey Austyn, I am probably not fully understanding, but the
assert
object of each test case takes a list of assertions. Can you provide multiple assertions there? That will use a single LLM call, but run multiple checks.
a
Thank you for your response, that will be helpful.
Hi again! Thanks again for your previous response, and making the assertions into a list has been very helpful, and our team has refactored our approach to include this, but we have still struggled with asserting on our model having multiple "links" in the LLM chain returned as JSON. For a simplified example, if our model responded with { "question": "How many cats", "category": "SQL", "query": "SELECT COUNT FROM cats;", "query_result": [[101]], "response": "Based on the data retrieved from our system, there are 101 cats." } Then it would be really great if there was a way to have an optional way in the promptfooconfig yaml to be able to select which key of the JSON response we are trying to assert on without having to write custom assertions that navigate into the json for all of our tests. That way it would save us on LLM calls, and also show up a lot prettier in the localhost page by showing only the output from the key being asserted on. an example of what our request might look like if implemented:
Copy code
assert:
          - type: equals
            value: SQL
             key: category
          - type: contains
             value: 101
             key: response
If this were to be implemented it would help our team out a whole lot and I suspect it would be used by others as well. Thanks for looking at this for us!
@ian Thanks for taking a look, I am wondering if you have had the chance to think about it further. Do you know if this is a feature that might be planned for implementation?
i
Hey @a.u.s.t.y.n., I've thought a bit about this and the way that I'm inclined to implement is by supporting the [
postprocess
](https://promptfoo.dev/docs/configuration/guide/#postprocessing) directive for assertions. It would look like this:
Copy code
yaml
assert:
  - type: equals
    value: SQL
    postprocess: output.category
This is consistent with other supported behavior to e.g. pick keys out of JSON objects. What do you think? Also, just to make sure that you're aware, you can achieve similar results with this:
Copy code
yaml
assert:
  - type: javascript
    value: output.category === 'SQL'
which is not too awful imo..
a
@ian Using the postprocess is exactly what we need! If you could do that it would be perfect. We did see the way to do the javascript assertion, but we are hoping to avoid them because our use case uses a wider variety of the prebuilt assertions, including llm-rubric. We are also hoping for the sake of readability to have the output from the model in promptfoo view just be the portion of the response we are asserting on.
i
understood. I think
postprocess
is a small lift. Let me see if I can get it done quickly 🙂
will be in the next release 🙂
a
Thank you so much, you are a hero to our project!
i
This has landed! Just noting that I renamed the field to
transform
, because
postprocess
was a bit too specific and doesn't quite make sense in the context of assertions 🤔 It is documented here:
a
Incredible, thank you so much!
8 Views