https://promptfoo.dev/ logo
Evaluate existing full conversation
# ❓|questions
c
Instead of evaluating the next llm output, i want to evaluate on the entire conversation history. What is the best way to set it up? 1. I don't want a provider to simluate the llm output 2. I want the assertion be evaluating based on the entire conversation history. Example assertion I want to have 1. check if assistant messages includes special punctuation, such as asterisk 2. check if the assistant tries to complete the sentence for the user. If user has an incomplete sentence, whether the assistant finish the rest of the sentence for the user 3. check if the assistant asks the same question multiple times
t
@Will [Promptfoo] @shuo0446
w
What provider are you using?
c
I don’t really want a provider to make an output. I just want to run tests against the existing conversation history
w
I would define each message in your conversation as a test case (alongside the relevant per-message assertion(s)), a
defaultTest
block if you have any assertions which should be applied to all messages, and then use the
echo
provider
For the
prompts
block, you could do something like
'{{message}}'
and then within your test cases use
vars.message
c
Nice, I think this is exactly what i need. Thank you
w
No problem! LMK if this doesn't work or you need more help
c
Hmm, I have a model-graded-qa assertion to check if the assistant asks the same questions multiple times. Since echo will repeat the input and output, the eval will falsely think the assistant ask the same question because input and output contains the same question. Is there other built-in alternative that fit this scenario? or will it make sense if i create a custom python provider that output an empty string?
w
Try this (replace the dummy conversation history)
Copy code
prompts:
  - "{{ messages | dump }}"

providers:
  - echo

defaultTest:
  assert:
    - type: model-graded-closedqa
      value: The assistant message does not include special punctuation, such as asterisks.
    - type: model-graded-closedqa
      value: The assistant does not try to finish incomplete sentences for the user.
    - type: model-graded-closedqa
      value: The assistant does not ask the same question multiple times.

tests:
  - vars:
      # See https://www.promptfoo.dev/docs/configuration/chat/
      messages:
        - role: system
          content: Respond as a pirate
        - role: user
          content: Who founded Facebook?
        - role: assistant
          content: Mark Zuckerberg
        - role: user
          content: Did he found any other companies?
c
Thank you. It seems like I made the assert prompt too complicated when it can as simple as you did 😅
w
It takes trial and error to get right 😄