https://promptfoo.dev/ logo
_conversation is weirdly available across multiple...
# ❓|questions
p
I have a setup where there are 10s of different test.yaml files along with multi turn chat prompt where i use _conversations (as per https://github.com/promptfoo/promptfoo/blob/main/examples/multiple-turn-conversation/prompt.json ) but i am seeing that the _conversation is not restricted to 1 test file, i want it to reset for each new test yaml file. i want to simulate 10 diff scenarios of the conversation (where user says diff things and then test the entire followup conversation based on that).
@ian is there a way to reset the value of the main/global _conversation variable?
I was thinking of have a var
newConversation= true
in my test vars in the first message per conversation. And then some code like given in this picture. But the problem is that the
_conversation
is getting set to
[]
only for that particular test (i.e a message in multi turn) but i want to reset it globally. (so that in the next message for the same thread the first msg is present not all the previous messages). https://cdn.discordapp.com/attachments/1223654806963486821/1223856905903149158/image.png?ex=661b6090&is=6608eb90&hm=ef338ea4c0519d52ea3d2f8e25ebec504e6b82d786a87865290d46caf050adb9&
Copy code
[
    {
      "role": "system",
      "content": "{{system_message}}"
    },

    {% if newConversation %}
        {% set _conversation = [] %}
    {% endif %}

    {% for completion in _conversation %}
      {
        "role": "user",
        "content": "{{ completion.prompt[completion.prompt.length - 1].content }}"
      },
      {
        "role": "assistant",
        "content": "{{ completion.output }}"
      },
    {% endfor %}
    {
      "role": "user",
      "content": "{{ question }}"
    }
  ]
In test format if you wanna suggest an edit easily
i
Unfortunately I don't have a solution for you because there isn't a way to reset the conversation variable globally 😦 you may just have to run them as separate evals for now There is some ongoing work in this area to make conversations a first-class concept:
p
ohh 😦
and just for added context, that PR is on the
threads
feature of OAI, i dont want to lock into that, so i use the chat api (with my code providing the chat history).
but yeah, if you can do anything for allowing to access the global _conversation it will be very helpful
i
understood 🙂 I would like to make this feature not
threads
specific, but for anyone who wants to maintain a conversation history
p
thank you
@ian is it impossible to do this OR just not priority? coz if you can give some pointers/directions, i might be able to crack this.
i
definitely not impossible 🙂 I'm hoping the author of will be able to contribute this (we want to make it so that it's not OAI-assistant specific). If he doesn't follow up, happy to work with you on it!
m
Hey @ian wanted to double check that this is still not available? I saw the PR wasn't merged but wanted to double check
i
that pr isn't landed but there's a separate feature called
storeOutputAs
that lets you store LLM outputs as variables that can be used in subsequent tests. Depending on your use case, it may help. Docs here:
m
Hmm, I think that could potentially work but might be easier just copy pasting than keeping track of all the outputs. I basically want to have the same prompt and then different conversations and tests for those conversations. So one conversation where the end user enters everything correctly Another test where the conversation is not as striaght forward and end user enters a not valid value. and so on. To me that sounds like having each as a different config file is the path forward. Is that correct?
i
yeah, for now I would recommend splitting the configs 🙂
2 Views