mikelacher
04/25/2024, 2:52 PM--verbose, it looks like generationConfig is empty.
Here's my promptfooconfig.yaml:
description: 'vertex test'
prompts:
- Tell me a joke about {{input}}
providers:
- id: vertex:gemini-1.5-pro-preview-0409
config:
generationConfig:
temperature: 2
safetySettings:
- category: HARM_CATEGORY_HARASSMENT
threshold: BLOCK_NONE
- category: HARM_CATEGORY_DANGEROUS_CONTENT
threshold: BLOCK_NONE
- category: HARM_CATEGORY_HATE_SPEECH
threshold: BLOCK_NONE
- category: HARM_CATEGORY_SEXUALLY_EXPLICIT
threshold: BLOCK_NONE
tests:
- vars:
input: dogs
- vars:
input: cats
I run with this command:
npx promptfoo@latest eval --no-cache --verbose
Then I see from the --verbose logs that this is what's being sent to Vertex. generationConfig is empty:
{
"contents": {
"role": "user",
"parts": {
"text": "Tell me a joke about dogs"
}
},
"generationConfig": {},
"safetySettings": [
{
"category": "HARM_CATEGORY_HARASSMENT",
"threshold": "BLOCK_NONE"
},
{
"category": "HARM_CATEGORY_DANGEROUS_CONTENT",
"threshold": "BLOCK_NONE"
},
{
"category": "HARM_CATEGORY_HATE_SPEECH",
"threshold": "BLOCK_NONE"
},
{
"category": "HARM_CATEGORY_SEXUALLY_EXPLICIT",
"threshold": "BLOCK_NONE"
}
]
}
Is this a bug or am I configuring it wrong? I'm admittedly a bit of a yaml dummy.ian
04/25/2024, 6:35 PMtemperature are taken from the top-level config object rather than nested in generationConfig
https://github.com/promptfoo/promptfoo/blob/main/src/providers/vertex.ts#L196
This does not match the documentation 😱 Sorry about that
I'll get this fixed, but basically you can work around this by just doing:
yaml
config:
temperature: 2mikelacher
04/25/2024, 6:39 PMian
04/26/2024, 12:37 AM