Tem
04/17/2024, 10:20 AMpeanutshawn
04/18/2024, 6:37 PMTem
04/18/2024, 9:12 PMTem
04/18/2024, 9:25 PMpiyush_63987
04/22/2024, 11:19 AMmikelacher
04/22/2024, 8:49 PMTem
04/23/2024, 3:23 AMjimd
04/23/2024, 6:50 PMswiecki
04/23/2024, 8:46 PMtypescript
const client = Instructor({
client: oai,
mode: "TOOLS",
})
const ExampleSchema = z.object({
size: z.string().describe("size"),
weight: z.string().describe("weight"),
})
return await client.chat.completions.create({
messages: [
{
role: "user",
content: `${inputString}`,
},
],
model: "gpt-4-turbo-2024-04-09",
response_model: {
schema: ExampleSchema,
name: "Physical_Info",
},
})
}
I'm hoping to use this as a provider basically and pass different input strings to it that are known cases and assert that certain things are true. Yes you can parse out specific JSON keys in asserts, thats NOT the issue- the issue is should I be doing one provider per yaml file since I'm not looking to run all tests against all providers. Is there a way I can use this function to run the tests but also keep the nice html output? Thanks!Will
04/24/2024, 12:18 AMQuestionMark1986
04/24/2024, 8:55 AMWill
04/24/2024, 2:50 PMresponse_format: {
type: 'json_object',
schema: {
properties: { tags: { title: 'tags', type: 'string' } },
required: ['tags'],
title: 'GetTags',
type: 'object',
},
},
I believe I'll need to set up a custom provider, but what's the best way for me to pass in this schema from a yaml file? Is there a way to do it in the current version of promptfoo or does this require an officially supported provider? Thanks!Henrique Coelho
04/24/2024, 3:44 PMmikelacher
04/25/2024, 2:52 PM--verbose
, it looks like generationConfig is empty.
Here's my promptfooconfig.yaml:
description: 'vertex test'
prompts:
- Tell me a joke about {{input}}
providers:
- id: vertex:gemini-1.5-pro-preview-0409
config:
generationConfig:
temperature: 2
safetySettings:
- category: HARM_CATEGORY_HARASSMENT
threshold: BLOCK_NONE
- category: HARM_CATEGORY_DANGEROUS_CONTENT
threshold: BLOCK_NONE
- category: HARM_CATEGORY_HATE_SPEECH
threshold: BLOCK_NONE
- category: HARM_CATEGORY_SEXUALLY_EXPLICIT
threshold: BLOCK_NONE
tests:
- vars:
input: dogs
- vars:
input: cats
I run with this command:
npx promptfoo@latest eval --no-cache --verbose
Then I see from the --verbose logs that this is what's being sent to Vertex. generationConfig is empty:
{
"contents": {
"role": "user",
"parts": {
"text": "Tell me a joke about dogs"
}
},
"generationConfig": {},
"safetySettings": [
{
"category": "HARM_CATEGORY_HARASSMENT",
"threshold": "BLOCK_NONE"
},
{
"category": "HARM_CATEGORY_DANGEROUS_CONTENT",
"threshold": "BLOCK_NONE"
},
{
"category": "HARM_CATEGORY_HATE_SPEECH",
"threshold": "BLOCK_NONE"
},
{
"category": "HARM_CATEGORY_SEXUALLY_EXPLICIT",
"threshold": "BLOCK_NONE"
}
]
}
Is this a bug or am I configuring it wrong? I'm admittedly a bit of a yaml dummy.Will
04/26/2024, 2:32 AMproviders:
- id: openai:chat:mixtral-8x7b-32768
config:
apiBaseUrl: https://api.groq.com/openai/v1
apiKeyEnvar: GROQ_API_KEY
- id: openai:chat:llama2-70b-4096
config:
apiBaseUrl: https://api.groq.com/openai/v1
apiKeyEnvar: GROQ_API_KEY
Looks like the model name is passed in via the id
. Totally fine with me, but I just want to be able to rename the column to show groq somewhere, otherwise it gets pretty confusing when comparing providers because the columns all show openai:chat:model
Is there a way to rename this? Tbh it does feel a bit awkward to use the id
field to handle the model.Will
04/26/2024, 11:13 PMprompts:
- prompts/few-shot-json.js
- prompts/true-false-json.js
providers:
- id: openai:chat:llama3-70b-8192
label: 'GROQ-LLAMA-70B'
config:
temperature: 0
apiBaseUrl: https://api.groq.com/openai/v1
apiKeyEnvar: GROQ_API_KEY
response_format: { type: 'json_object' }
- id: openai:chat:llama3-8b-8192
label: 'GROQ-LLAMA-8B'
config:
temperature: 0
apiBaseUrl: https://api.groq.com/openai/v1
apiKeyEnvar: GROQ_API_KEY
response_format: { type: 'json_object' }
I tried following the docs instruction: To reference a specific function in your prompt file, use the following syntax: filename.js:functionName:
, but this didn't work. The code runs without error but 0 evals are run.
prompts:
- prompts/few-shot-json.js:few-shot-prompt
- prompts/true-false-json.js:true-false-prompt
providers:
- id: openai:chat:llama3-70b-8192
label: 'GROQ-LLAMA-70B'
prompts:
- prompts/few-shot-json.js:few-shot-prompt
config:
temperature: 0
apiBaseUrl: https://api.groq.com/openai/v1
apiKeyEnvar: GROQ_API_KEY
- id: openai:chat:llama3-8b-8192
label: 'GROQ-LLAMA-8B'
prompts:
- prompts/true-false-json.js:true-false-prompt
config:
temperature: 0
apiBaseUrl: https://api.groq.com/openai/v1
apiKeyEnvar: GROQ_API_KEY
When I remove the prompts:
at the top, it errors out and says
throw new Error(`There are no prompts in ${JSON.stringify(promptPathOrGlobs)}`);
^
Error: There are no prompts in {}
at readPrompts (/Users/wl/.nvm/versions/node/v18.17.0/lib/node_modules/promptfoo/dist/src/prompts.js:273:15)
Can anyone provide an example? That would be super helpful. Thanks!Henrique Coelho
04/29/2024, 3:27 PMRajesh Ch
04/29/2024, 3:38 PMHenrique Coelho
04/29/2024, 4:33 PMDieter
04/30/2024, 7:33 AMknathaannnn
04/30/2024, 8:38 PMvitmont
04/30/2024, 9:27 PMtest.yaml
file, but it would be nice to supplt an additional vars on the command line that get read into the provider that are not defined in the test.yaml
file so it is configurable on the fly without editting the test definition.
For example, if we have
yaml
vars:
a: "some var"
assert:
...
We can pull the vars into the provider (python) similarly to
py
kwargs = json.loads(sys.argv[3])["vars"]
var_a = kwargs["a"] # "some var"
But could additionally supply another var such as
bash
promptfoo eval --var b="somevalue"
where we could then
py
kwargs = json.loads(sys.argv[3])["vars"]
var_a = kwargs["a"] # "some var"
var_b = kwargs.get("a", "default_value")
Mariuz
05/01/2024, 4:52 PMdavidl
05/01/2024, 11:10 PMEdgar
05/02/2024, 7:33 PMjimd
05/02/2024, 7:47 PMWill
05/03/2024, 5:54 PMjoemiller88
05/06/2024, 4:06 PMpromptfoo eval -c router/*
And getting this error
/node/v20.5.0/lib/node_modules/promptfoo/dist/src/util.js:193
throw new Error(Unsupported configuration file format: ${ext}
);
Note - I am using javascript config files, so I assume that perhaps this has only been tested with the YAML format and is not setup to work with javascript config files?knathaannnn
05/08/2024, 8:25 PMandrew
05/09/2024, 1:49 AMsrc
|_ prompts
|_ prompt-that-classifies-utterance
| |_ prompt.ts
| |_ tests
| |_ output
| |_ testExecutor.ts
|
|_ prompt-that-extracts-dates
|_ prompt.ts
|_ tests
|_ output
|_ testExecutor.ts
I could cd into each /tests directory and run promptfoo view .
which would only show me evals associated with a specific prompt. This was very neat.
Now I don't see how I can distinguish between all of the prompts in my project since they all get saved to a single sqllite database.
Is there a legitamate path forward with JSON, or is the sqlite the only way?