https://promptfoo.dev/ logo
Testing multi-line system prompt (which has " etc...
# ❓|questions
p
Context -- I have a big system prompt that has 100+ lines and multiple places with double quotes etc. And has around 10 variables (which i want to test for with different values). And I want to test this with multi-turn conversations. Main question -- How to structure the main
prompts
section for the promptfooconfig.yaml ?? Option 1 - Should i make a json prompt file like https://github.com/promptfoo/promptfoo/blob/main/examples/multiple-turn-conversation/prompt.json ? Iif yes then how to supplement other 10 vars within this prompt AND how to provide the big system prompt with proper escaping in nunjunks? Option 2 - Should i make js prompt file? then how will i use nunjunks for the _conversation loop which is needed for multi turn convo?
@ian any examples or suggestions for this?
i
Yes, I would make this a JSON prompt file. - A multiline system prompt will be automatically escaped in nunjucks - You can either manually create the conversation in the prompt, or you can create vars with messages and substitute them. Here's an example of how to create a list of messages as a var:
p
can you give an example of
A multiline system prompt will be automatically escaped in nunjucks
with a json file, coz in my case the editor keep throwing error if i pase a multi line prompt in the json file and that too a prompt which has multiple double qoutes in the prompt string itself
i
so I've gone and tried myself, and found that the nunjucks control logic like
{% for message in previous_messages %}
cause the file to not be interpreted as JSON, which in turn prevents the auto-escaping behavior. The workaround is to use the built-in nunjucks
dumps
filter, which converts the variable to JSON. Here's an example:
Copy code
js
[
  {
    "role": "system",
    "content": {{ system_message | dump }}
  },
  {% for message in previous_messages %}
    {% for role, content in message %}
      {
        "role": {{ role | dump }},
        "content": {{ content | dump }}
      },
    {% endfor %}
  {% endfor %}
  {
    "role": "user",
    "content": {{ question | dump }}
  }
]
In this example,
system_message
is imported as a multiline text file and is properly quoted/escaped.
p
ok thanks