Hello, I have problems concerning the simulated_user while using an AzureOpenAI provider. I want to test my assistant's prompt which requires multi-turn conversation. The functionality of the "simulated_user" provider seems to perfectly match my use case. But what ever I try, no conversation takes place between the assistant and the simulated user. There is just one greeting message from the assistant. Tests that assert certain information further into the conversation obviously fail because the one greeting message does not meet the requirements. I already made sure that maxTurns is set >5 to allow several messages and that the same AzureOpenAI provider is set for the simulated user as well: defaultTest: provider: id: promptfoo:simulated-user config: maxTurns: 8 options: provider: id: azure:chat:gpt-4.1-mini I would really appreciate your help.

Can you share your full config?

Of course. This is my promptfooconfig.yaml: description: "LeadBooster Evaluation" prompts: # Set prompts in external files - "You are a telephone assistant for a doctor's office. Speak in a friendly manner, ask questions, and briefly summarize what has been agreed upon. IMPORTANT: Only say ‘###STOP###’ if (a) the issue has been resolved AND (b) the name and telephone number have been recorded AND (c) the caller has confirmed that the call can be ended. Do NOT say ‘###STOP###’ in the first 3 turns." providers: - id: "azure:chat:gpt-4.1-mini" label: "azure:chat:gpt-4.1-mini [LOWTEMP=0.0]" config: temperature: 0.0 max_tokens: 200 # Use the same model again to evaluate the influcence of parameters (use labels to distinguish used parameters) - id: "azure:chat:gpt-4.1-mini" label: "azure:chat:gpt-4.1-mini [HIGHTEMP=1.0]" config: temperature: 1.0 max_tokens: 200 # Set Properties for all tests to avoid repetition defaultTest: provider: id: "promptfoo:simulated-user" config: maxTurns: 10 options: provider: id: "azure:chat:gpt-4.1-mini" # Set general checks that need to pass for all tests assert: - type: llm-rubric value: make sure that the assistant is friendly and professional # Simulate custom user interactions tests: - vars: instructions: "You call to book an appointment for a routine check-up next week. You are only available Monday to Friday between 9 a.m. and 12 p.m. You want to know if an earlier appointment is possible. Answer concisely but naturally; ask questions if anything is unclear." -------------------------------------------------------------- In the same directory is a .env file: AZURE_API_KEY=XXX AZURE_API_HOST=XXX.openai.azure.com AZURE_DEPLOYMENT_NAME=gpt-4.1-mini

Thanks! I'll take a look at this and get back to you

Thank you in advance!

Having the same issue. I reported the same issue a couple of months ago: (https://github.com/promptfoo/promptfoo/issues/3214) which was fixed afterwards On first glance it seems as if the simulatedUser provider has been rewritten and that this problem is reintroduced because of that rewrite.

I've opened a PR with the fix: https://github.com/promptfoo/promptfoo/pull/5348. Essentially

defaultTest.provider

is not being correctly read.

Thank you very much for fixing this quick. Using the simulated user will be a game changer for me! When do you expect the changes to go live? And do I need to manually update promptfoo at this point?

If you want to use it now (and you are have installed it via npm) you can do what I did: 1. open the promptfoo node module 2.open

dist/src/providers/simulatedUser.js

3. change the agentResponse line to the following:

const prompt = context.prompt;
const agentVars = context.vars;
const agentInstructions = (0,
templates_1.getNunjucksEngine)().renderString(
    prompt.raw,
    agentVars
);

agentResponse = await this.sendMessageToAgent(
    [
        {
            role: "assistant",
            content: agentInstructions,
        },
        ...messagesToUser,
    ],
    context.originalProvider,
    context
);

that could be an option in the meantime while waiting for Wills PR to be merged

The change has been merged into our

main

branch, but I am not sure when the next release will be – in the interim you could set the simulated user provider on each test's

provider

property

Hello Arian, thank you so much for pointing this out and providing a temporary fix. I didn't even know I had this problem, till you wrote this. At first I only had the problem, that I could not set the simulated-user provider as the default in defaultTest (multi-turn-conversation not working). When you first sent me your old ticket I thought it was related to simulated-user but not the same as my problem. But when using the workaround from Will, by setting the provider for each test and the multi-turn conversation worked right, I saw that my assistant does not use its prompt. In the meantime the fix for the defaultTest-provider was released. It helped me with my first problem, but didn't solve my second problem - but your approach did! I did not even ask about this specifically, but you saw that I would eventually run into the same problem and provided help. Thank you. You really made my day!

Hi @elias_m2m_54456 , sorry for the radio silence. I will take a look into that this week for you and report back.

Hello Will, any updates on this? I'm still using Version 0.117.10 with the workaround. It would be nice to be able to use the current version with all the new features and fixes.