johnjohnjohnjohnjohn
06/21/2024, 6:12 PMprompts:
- '{{messages | dump}}'
defaultTest:
options:
provider:
id: bedrock:anthropic.claude-3-sonnet-20240229-v1:0
config:
region: 'us-east-1'
providers:
- id: 'http://localhost:8000/v2/promptfoo'
config:
method: 'POST'
headers:
'Content-Type': 'application/json'
body:
version: '{{version}}'
prompt: '{{prompt}}'
model: '{{model}}'
responseParser: '{ output: json.body }'
tests:
- vars:
version: ['latest']
model: ['SONNET_3', 'SONNET_35', 'HAIKU_3']
messages:
- role: 'user'
content: 'whats up boss'
Given the above, the generated payload for is invalid JSON and dies trying to parse itself:
{"method":"POST","headers":{"Content-Type":"application/json"},"body":{"version":"latest","prompt":"[{"role":"user","content":"whats up boss"}]","model":"HAIKU_3"},"responseParser":"{ output: json.body }","basePath":""}
(note that the prompt array has unescaped quotes around it!)
I thought maybe I could fix this by seting prompt: {{prompt}} without any quotes, which does indeed result in a http request, but with the following:
json
{"method":"POST","headers":{"Content-Type":"application/json"},"body":{"version":"latest","prompt":{"[object Object]":null},"model":"SONNET_3"},"responseParser":"{ output: json.body }","basePath":""}
(note the [object Object] where the prompt should be)
Am I missing something totally obvious here, or is this an issue with the HTTP provider?johnjohnjohnjohnjohn
06/24/2024, 6:30 PMconfig object , then do the nunjucks replacement, and then try to JSON.parse it back to get the fully rendered config object. I putzed around trying to think of how to fix this, but ended up just working around it by writing a custom provider mjs that just makes a fetch request itself 👍ian
06/26/2024, 6:22 AMmichaelmichaelmichael
07/16/2024, 1:23 AM