https://promptfoo.dev/ logo
Powered by
Bug - prompt from python not loaded under simulate...
# ❓｜questions
c
I think there's a bug when prompts are loaded from python + doing 
promptfoo:simulated-user
tests. Promptfoo is reading the file content as the prompt, not the function return. When the config and Python are the following,
Copy code
# config.yaml
providers:
  - openai:gpt-5-mini

prompts:
  - file://generate_prompt.py:create_prompt

defaultTest:
  provider:
    id: "promptfoo:simulated-user"
    config:
      maxTurns: 2

tests:
  - vars:
      instructions: Learn how to use react.js

# -----------------------------------------
# generate_prompt.py

from system_prompt import my_prompt

def create_prompt(context):
    return my_prompt

# -----------------------------------------
# system_prompt.py (storing full prompt in another file)

my_prompt = "Always respond in 10 words or less"
Assistant doesn't follow 
my_prompt
. Full response in image 1.png If I update 
generate_prompt.py
to the following, assistant response will be <= 10 words
Copy code
# generate_prompt.py
from system_prompt import my_prompt

def create_prompt(context):
    return "Always respond in 10 words or less"
Assistant will respond in 10 words. Full response in image 2.png. But I believe The prompt works fine when - loaded inline in yaml - loaded by json - there's no simulated provider for the tests. https://cdn.discordapp.com/attachments/1487832982466199666/1487832982839361836/Screenshot_2026-03-29_at_11.05.31_PM.png?ex=69ca93f1&is=69c94271&hm=3740559e5a811653ca0bf4403033ac9948a1d17ebcaf9da9dfc3e4d664e0f28b& https://cdn.discordapp.com/attachments/1487832982466199666/1487832983506260050/Screenshot_2026-03-29_at_11.06.59_PM.png?ex=69ca93f1&is=69c94271&hm=56ff17d097cf03825e0b138ae8845fb762af28215789fc190f69c53610f5798a&
When 
generate_prompt.py
is the following, even though the create_prompt returns empty string, assistant still follows the 10 words policy,
Copy code
# generate_prompt.py
from system_prompt import my_prompt

"Always respond in 10 words or less"
def create_prompt(context):
    return ""
https://cdn.discordapp.com/attachments/1487832982466199666/1487833613809745990/Screenshot_2026-03-29_at_11.18.23_PM.png?ex=69ca9488&is=69c94308&hm=e398fd7fd5794ea146af55d2ce9bd8d1920c00544d974546671c6fee14a70860&
i
c
Thank you!
Oh also, I think 
model-graded-closedqa
on simulated user has a bug too. (Not using prompt from python). I get "Model grader produced a malformed response:". It seems like the eval starts its own conversation rather than following the model-graded-closedqa system prompt. Is this something your team can control? Another question - should I use llm-rubric or model-graded-closedqa in majority of the case?
Copy code
# config.yaml
providers:
  - openai:gpt-5-mini

prompts:
  - Always respond in 10 words or less

defaultTest:
  provider:
    id: "promptfoo:simulated-user"
    config:
      maxTurns: 3

tests:
  - vars:
      instructions: Learn how to use react.js
    assert:
      - type: model-graded-closedqa
        value: The assistant responds in 10 words or less
https://cdn.discordapp.com/attachments/1487832982466199666/1488098834671206530/Screenshot_2026-03-30_at_4.51.34_PM.png?ex=69cb8b89&is=69ca3a09&hm=cebe994f5fdaf3c9fdadaf33a5f4495b813a69050018e2813c28ee84f0ed32ae& https://cdn.discordapp.com/attachments/1487832982466199666/1488098835124322364/Screenshot_2026-03-30_at_4.51.41_PM.png?ex=69cb8b89&is=69ca3a09&hm=c0dd54de7b300f424c49649ada8f7665953e21eedbc67328e7d245199ea9780d&
i
will look into this... that said, I recommend`llm-rubric`, it is vastly more popular
2 Views
PreviousNext
Powered by