https://promptfoo.dev/ logo
assistant: evaluating a thread
# ❓|questions
j
Is it possible to evaluate a thread with promptfoo? Passing a thread to the provider config didn't work for me (as in the docs: https://www.promptfoo.dev/docs/guides/evaluate-openai-assistants#comparing-different-versions-of-the-same-assistant), but even if it worked I would like to test multiple threads, so passing it in the global config wouldn't be the right solution. It would be fantastic if it would be possible to declare messages as being part of the same thread - with evals possible on each message. Perhaps something like this is already possible; if not, I'd love to get some pointers on contributing this.
i
The tricky part is that the thread is stateful, right? So when a message is added, it's there forever. Does that mean the eval won't be useful the second time around?
j
Right now each message initiates a new thread. I'd love to do a follow up question. Ideally it would be great to give the assistant a history of a thread as part of the "prompt". But not sure that is possible with the API. The next best would be to ask a question, assert, and then ask the next question in the same thread.
n
Hello @jaapf , did you manage to solve this issue? I'm interested in doing something similar. Please, let me know 😁