https://promptfoo.dev/ logo
Possible to include prompt in promptfooconfig.yaml...
# ❓|questions
r
Is it possible to include the prompts within promptfooconfig.yaml, rather than linking to a prompts.txt file? When I save the config from the browser interface (using promptfoo view) it seems to have that, like in this example:
Copy code
env: {}
description: ''
providers:
  - id: openai:gpt-4
prompts:
  - |-
    [
        {
            "role": "system",
            "content": "You are an AI assistant."
        },
        {
            "role": "user",
            "content": "Below is a sentence in English. Please translate it into Pirate."
        }
    ]
  - |-
     [
        {
            "role": "system",
            "content": "You are an AI assistant."
        },
        {
            "role": "user",
            "content": "Another prompt..."
        }
    ]
However, when I paste that into my promptfooconfig.yaml file, I get an error (ENAMETOOLONG) -- it appears to be trying to use all that text as the filename for the prompts. Thanks!
k
.yaml:
Copy code
# This configuration runs each prompt through a series of example inputs and checks if they meet requirements.

prompts: [prompts.txt]
providers: [openai:gpt-3.5-turbo-0613]
tests:
  - description: prompt1
    vars:
      var1:
      - |-
            [
                {
                    "role": "system",
                    "content": "You are an AI assistant."
                },
                {
                    "role": "user",
                    "content": "Below is a sentence in English. Please translate it into Pirate."
                }
            ]  
  - description: prompt2
    vars:
      var1:
      - |-
          [
              {
                  "role": "system",
                  "content": "You are an AI assistant."
              },
              {
                  "role": "user",
                  "content": "Another prompt..."
              }
          ]
prompts.txt
Copy code
{{var1}}
Still have to use the prompts.txt file but the prompts are on var1 in the YAML file. Idk if it helps you, I would just use the text file for setting the prompts but Idk what are your reasons.
i
this is not supported currently in CLI usage, but it's something that I could add if you all think it'd be useful
r
Yes, this would be really handy for me if you're able to add it to the CLI usage, because it would let me keep everything needed to run a test in a single config file. Also, it could mean that the config file that is downloadable from the web interface could be re-used in the CLI without needing alteration. Thanks, @ian !
Thanks for that tip, @kevin427 -- I'm still looking for away to actually have the prompts stored in the config YAML, but this good to know.
i
ok, will implement this one shortly 🙂
r
Nice!! Thanks again, @ian !
Hi @ian , With that revision that supports raw text prompts, does it still work to reference a prompts.js from the promptfooconfig.yaml file like this?:
Copy code
prompts: [prompts.js:prompt1, prompts.js:prompt2]
For some reason, that no longer works for me.
i
hmm, let me look into this
Found the issue - this will be fixed in tonight's release
thanks for the report
r
That’s great, thanks!
2 Views