https://promptfoo.dev/ logo
struggling to setup HTTP Provider + multi-turn con...
# ❓|questions
j
👋 Here's my simplified test case -- you don't actually need a HTTP server listening/responding as promptfoo dies before it even makes the HTTP request 😦
Copy code
prompts:
  - '{{messages | dump}}'

defaultTest:
  options:
    provider:
      id: bedrock:anthropic.claude-3-sonnet-20240229-v1:0
      config:
        region: 'us-east-1'

providers:
  - id: 'http://localhost:8000/v2/promptfoo'
    config:
      method: 'POST'
      headers:
        'Content-Type': 'application/json'
      body:
        version: '{{version}}'
        prompt: '{{prompt}}'
        model: '{{model}}'
      responseParser: '{ output: json.body }'

tests:
  - vars:
      version: ['latest']
      model: ['SONNET_3', 'SONNET_35', 'HAIKU_3']
      messages:
        - role: 'user'
          content: 'whats up boss'
Given the above, the generated payload for is invalid JSON and dies trying to parse itself:
Copy code
{"method":"POST","headers":{"Content-Type":"application/json"},"body":{"version":"latest","prompt":"[{"role":"user","content":"whats up boss"}]","model":"HAIKU_3"},"responseParser":"{ output: json.body }","basePath":""}
(note that the prompt array has unescaped quotes around it!) I thought maybe I could fix this by seting
prompt: {{prompt}}
without any quotes, which does indeed result in a http request, but with the following:
Copy code
json
{"method":"POST","headers":{"Content-Type":"application/json"},"body":{"version":"latest","prompt":{"[object Object]":null},"model":"SONNET_3"},"responseParser":"{ output: json.body }","basePath":""}
(note the [object Object] where the prompt should be) Am I missing something totally obvious here, or is this an issue with the HTTP provider?
dug into this a bit -- it looks like we JSON.stringify the
config object
, then do the nunjucks replacement, and then try to JSON.parse it back to get the fully rendered config object. I putzed around trying to think of how to fix this, but ended up just working around it by writing a custom provider
mjs
that just makes a fetch request itself 👍
i
thanks for looking into this. Will get a fix in for the http povider
m
Thanks for reporting this. We rolled out a fix for this today in 0.71.1. Sorry it took us so long to resolve it.
2 Views