janricks.marn
02/25/2025, 12:37 AMian
02/27/2025, 7:21 PMdefaultTest
https://www.promptfoo.dev/docs/configuration/expected-outputs/model-graded/llm-rubric/#overriding-the-llm-graderDaina
08/13/2025, 1:52 PMWill [Promptfoo]
08/13/2025, 2:02 PMpromptfooconfig.yaml?Daina
08/13/2025, 2:28 PMDaina
08/13/2025, 2:34 PMDaina
08/13/2025, 2:39 PMpromptfoo eval --grader https)Will [Promptfoo]
08/13/2025, 5:26 PMprompts:
- "In 150 words or less, tell me about {{topic}}"
providers:
- id: https
config:
url: https://api.openai.com/v1/chat/completions
method: POST
headers:
Authorization: "Bearer {{env.OPENAI_API_KEY}}"
Content-Type: "application/json"
body:
{
"model": "gpt-4o",
"messages": [{"role": "user", "content": "{{prompt}}"}]
}
transformResponse: "json.choices[0].message.content"
tests: https://docs.google.com/spreadsheets/d/1WuxP728Bl7cG4JeSJFLWXKMiXDv72trkbXrGNN25I1Y/edit?usp=sharing
is working in the latest version of promptfoo.Will [Promptfoo]
08/13/2025, 5:28 PMdefaultTest or override the grader.Daina
08/14/2025, 1:13 PMDaina
08/14/2025, 3:17 PMWill [Promptfoo]
08/18/2025, 3:19 PMprompts:
- "In 150 words or less, tell me about {{topic}}"
providers:
- id: https
config:
url: https://api.openai.com/v1/chat/completions
method: POST
headers:
Authorization: "Bearer {{env.OPENAI_API_KEY}}"
Content-Type: "application/json"
body:
{
"model": "gpt-4o",
"messages": [{"role": "user", "content": "{{prompt}}"}]
}
transformResponse: "json.choices[0].message.content"
defaultTest:
options:
provider: https
tests: https://docs.google.com/spreadsheets/d/1WuxP728Bl7cG4JeSJFLWXKMiXDv72trkbXrGNN25I1Y/edit?usp=sharing
It's occurring because defaultTest.options.provider doesn't read https as a reference to the provider defined in the providers block but instead expects the configuration to be provided in full.Will [Promptfoo]
08/18/2025, 3:21 PMconfig will be the same across both, it may be easier to migrate the values into a YAML file and import that instead of having duplicate declarations. See https://www.promptfoo.dev/docs/configuration/guide/#import-providers-from-separate-files.Daina
08/18/2025, 8:55 PMWill [Promptfoo]
08/20/2025, 2:02 PMWill [Promptfoo]
08/20/2025, 9:57 PMapiBaseUrl parameter. Docs here: https://www.promptfoo.dev/docs/providers/openai/#configuring-parametersDaina
08/21/2025, 7:19 PMWill [Promptfoo]
08/21/2025, 8:19 PMapiHost insteadWill [Promptfoo]
08/21/2025, 8:19 PMapiHost: https://app.credal.----.gov/api/openaiWill [Promptfoo]
08/21/2025, 9:01 PM