https://promptfoo.dev/ logo
Struggling a bit with .js prompt files in the yaml
# ❓|questions
w
What I'd like to do: Run specific prompts per provider. This works, but both prompts are run on both providers:
Copy code
prompts:
  - prompts/few-shot-json.js
  - prompts/true-false-json.js

providers:
  - id: openai:chat:llama3-70b-8192
    label: 'GROQ-LLAMA-70B'
    config:
      temperature: 0
      apiBaseUrl: https://api.groq.com/openai/v1
      apiKeyEnvar: GROQ_API_KEY
      response_format: { type: 'json_object' }
  - id: openai:chat:llama3-8b-8192
    label: 'GROQ-LLAMA-8B'
    config:
      temperature: 0
      apiBaseUrl: https://api.groq.com/openai/v1
      apiKeyEnvar: GROQ_API_KEY
      response_format: { type: 'json_object' }
I tried following the docs instruction:
To reference a specific function in your prompt file, use the following syntax: filename.js:functionName:
, but this didn't work. The code runs without error but 0 evals are run.
Copy code
prompts:
  - prompts/few-shot-json.js:few-shot-prompt
  - prompts/true-false-json.js:true-false-prompt

providers:
  - id: openai:chat:llama3-70b-8192
    label: 'GROQ-LLAMA-70B'
    prompts:
      - prompts/few-shot-json.js:few-shot-prompt
    config:
      temperature: 0
      apiBaseUrl: https://api.groq.com/openai/v1
      apiKeyEnvar: GROQ_API_KEY
  - id: openai:chat:llama3-8b-8192
    label: 'GROQ-LLAMA-8B'
    prompts:
      - prompts/true-false-json.js:true-false-prompt
    config:
      temperature: 0
      apiBaseUrl: https://api.groq.com/openai/v1
      apiKeyEnvar: GROQ_API_KEY
When I remove the
prompts:
at the top, it errors out and says
Copy code
throw new Error(`There are no prompts in ${JSON.stringify(promptPathOrGlobs)}`);
              ^

Error: There are no prompts in {}
    at readPrompts (/Users/wl/.nvm/versions/node/v18.17.0/lib/node_modules/promptfoo/dist/src/prompts.js:273:15)
Can anyone provide an example? That would be super helpful. Thanks!
i
The format for
prompts
is a little different if you want to reference them from within providers. Here's an example:
basically you have to give them a name in the top-level
prompts
section, and then reference that name in
providers.prompts
w
@ian that's actually the first thing I tried, but I kept getting this error, so I assumed .js prompts had to be named within the module.exports like the docs show.
Copy code
/Users/wl/.nvm/versions/node/v18.17.0/lib/node_modules/promptfoo/dist/src/util.js:247
        if (relativePath.startsWith('file://')) {
                         ^

TypeError: relativePath.startsWith is not a function
This code below throws the error above. Do you see something wrong with the way I'm doing it here?
Copy code
prompts:
  - prompts/true-false.js: true-false-prompt

  - prompts/many-shot.js: many-shot-prompt

providers:
  - id: openai:chat:llama3-70b-8192
    prompts:
      - true-false-prompt
    label: 'GROQ-LLAMA-70B'
    config:
      temperature: 0
      apiBaseUrl: https://api.groq.com/openai/v1
      apiKeyEnvar: GROQ_API_KEY
      response_format: { type: 'json_object' }
  - id: openai:chat:llama3-8b-8192
    prompts:
      - many-shot-prompt
    label: 'GROQ-LLAMA-8B'
    config:
      temperature: 0
      apiBaseUrl: https://api.groq.com/openai/v1
      apiKeyEnvar: GROQ_API_KEY
      response_format: { type: 'json_object' }
i
yep I see the problem!
prompts
should not be an array with this format. You should remove the `- `:
Copy code
yaml
prompts:
  prompts/true-false.js: true-false-prompt

  prompts/many-shot.js: many-shot-prompt
This bit me the other day actually, and the next release includes a better error message
w
ahhh got it. thanks @ian!
2 Views