https://promptfoo.dev/ logo
Many variables don't get used by simulated user (m...
# questions
e
Hello, I have a problem when using many variables to dynamically adjust the simulated-user's instructions. I wrote an instruction template (in the promptfooconfig.yaml) including many variable placeholders. The simulated user should only use the values of the variables to answer questions about this topic. In every test I set the variables according to the test. When viewing the results, the variable values are shown correctly, but the simulated-user seems to only notice the first few variable values (<6). Because he does not have values for the other placeholders he says he is not sure about the certain topic. Here is a simple dummy usecase: ... defaultTest: provider: id: promptfoo:simulated-user config: maxTurns: 30 options: provider: id: azure:chat:gpt-4.1-mini vars: instructions: | You are a young wizard who wants to go to Hogwarts. The sorting hat asks you a few questions to find out, which house you fit best. Only respond if you have been asked a specific question. Answer questions using only the following information: prename: {{prename}} surname: {{surname}} date of birth: {{date_of_birth}} favourite color: {{favourite_color}} favourite food: {{favourite_food}} favourite city: {{favourite_city}} hobby: {{hobby}} number of siblings: {{number_of_siblings}} tests: - description: "Harry" vars: prename: Harry surname: Potter date_of_birth: 31 July 1980 favourite_color: Red favourite_food: Treacle tart favourite_city: London hobby: Quidditch / flying number_of_siblings: 0 ... In all tests the first 5 questions could be answered using the variable values. But the last questions (favourite city, hobby, number of siblings) were not answered by the wizards or they said, they are not sure about it. Is there a maximum number of variables you could use? Is there a recommended workaround for this?
t
@[Promptfoo] Faizan is there a maximum number of variables or is that determined by something else?
u
I don't think there's any limit on number of vars- wonder if @Will [Promptfoo] (who implemented simulated user) might know? I wonder if this is a context limit issue, though it doesn't seem like it would be 🤔
e
I currently use a workaround and combine several variables into one to have a maximum of 5 variables total, but I wonder if there is another way.