Hi everyone 👋 I’m testing a medical chatbot agent that starts from a symptom (e.g. “I have stomach pain”), then asks several related questions, and finally recommends a suitable clinic. The problem is that each test run may have different question wording or order, so I can’t predefine all user inputs in advance. I’d like to dynamically provide user replies based on the agent’s last question — for example, if the agent asks about pain_location, I return the predefined answer for that property. Is there any recommended way in Promptfoo to handle this kind of dynamic multi-turn input-output flow? Thanks!

I have a similar use case and I use the simulated-user provider for this. It lets to simulate a user via another llm that can be prompted using instructions. Therefore it is able to handle different question wording or order. I prefer setting the base instruction prompt in default test with variable placeholders and set the variables in the tests. Here a basic example of what the default test var section could look like: vars: instructions: | Your name is Mr. Wright and you have medical problems. You reach out to a medical assistant to get a reccommendation for the suitable clinic that can help you. Only respond when you are explicitly asked a question! Respond appropriately to the assistant's questions and answers. Respond in naturally formulated sentences. Answer questions exclusively with the following information: Symptoms: {{symptoms}} https://www.promptfoo.dev/docs/providers/simulated-user/

Hi! I am adding support for seeing an initial conversation. Does this look helpful to you? https://github.com/promptfoo/promptfoo/pull/6090/files

Thank you very much! This saves so much time and tokens. Now I am able to specifically test a small section of a conversation flow, while having the context of the whole conversation. Thank you 🙂