https://promptfoo.dev/ logo
Using HTTP/HTTPS API Provider : Error in model-gra...
# ❓|questions
j
I am using an HTTPS API provider in Promptfoo and want to use model-graded metrics. However, I encountered the following error: Error: Invariant failed: Expected HTTP provider https to have a config containing {body}, but instead got {} When I checked using --verbose, I noticed that Promptfoo defaults to the OpenAI provider for llm-rubric and other model-graded metrics. Can you help me how to properly set-up my HTTP API provider that can use model-graded metrics? PS: it works fine on deterministic metrics https://cdn.discordapp.com/attachments/1343743556505960459/1343743556963274865/image.png?ex=689dd875&is=689c86f5&hm=440090161862c223fc7b7b114ab478ba40d03f314ff680bf16a342c23ec1a435& https://cdn.discordapp.com/attachments/1343743556505960459/1343743557244026992/image.png?ex=689dd875&is=689c86f5&hm=c4f937d9370e0ca412d8fa60d282b0799728b16ac2faa738253475cd87b60dac&
i
for model-graded metrics, you'll have to override the grader in
defaultTest
https://www.promptfoo.dev/docs/configuration/expected-outputs/model-graded/llm-rubric/#overriding-the-llm-grader
d
Hey Marco I'm having this exact issue. Did you ever figure out how to fix this? Nothing I have tried using the comments or links I've seen on here has worked. I understand that I need to override the grader, I think I'm just not doing something right
w
Hi @Daina – can you share your
promptfooconfig.yaml
?
I should say that for the file yaml file where I was trying to read the tests from a spreadsheet I had been trying the CLI method to override the grader (
promptfoo eval --grader https
)
w
Copy code
prompts:
  - "In 150 words or less, tell me about {{topic}}"

providers:
  - id: https
    config:
      url: https://api.openai.com/v1/chat/completions
      method: POST
      headers:
        Authorization: "Bearer {{env.OPENAI_API_KEY}}"
        Content-Type: "application/json"
      body:
        {
          "model": "gpt-4o",
          "messages": [{"role": "user", "content": "{{prompt}}"}]
        }
      transformResponse: "json.choices[0].message.content"

tests: https://docs.google.com/spreadsheets/d/1WuxP728Bl7cG4JeSJFLWXKMiXDv72trkbXrGNN25I1Y/edit?usp=sharing
is working in the latest version of promptfoo.
You should be able to read the tests from your Google Sheet and don't have to set
defaultTest
or override the grader.
d
Thank you for the quick follow up. Yes, that is working, but I'm working for a US government agency and need to use a platform called [Credal AI](https://www.credal.ai/) as my endpoint, so I need to use "https://app.credal.agencyname.gov/api/openai/v1/chat/completions" instead of https://api.openai.com/v1/chat/completions. I need to use Credal AI as the grader too, so that's why I need to override the grader. Credal is like a middleware that enables us to meet security requirements. Sorry I had scratched out the url before as I'm not sure I should be sharing the agency I'm working for.
Was something wrong with the syntax of my yaml or the way I was using the CLI method?
w
Thanks for the additional info – I was able to replicate your original error with the following:
Copy code
prompts:
  - "In 150 words or less, tell me about {{topic}}"

providers:
  - id: https
    config:
      url: https://api.openai.com/v1/chat/completions
      method: POST
      headers:
        Authorization: "Bearer {{env.OPENAI_API_KEY}}"
        Content-Type: "application/json"
      body:
        {
          "model": "gpt-4o",
          "messages": [{"role": "user", "content": "{{prompt}}"}]
        }
      transformResponse: "json.choices[0].message.content"

defaultTest:
  options:
    provider: https

tests: https://docs.google.com/spreadsheets/d/1WuxP728Bl7cG4JeSJFLWXKMiXDv72trkbXrGNN25I1Y/edit?usp=sharing
It's occurring because
defaultTest.options.provider
doesn't read
https
as a reference to the provider defined in the
providers
block but instead expects the configuration to be provided in full.
If the provider's
config
will be the same across both, it may be easier to migrate the values into a YAML file and import that instead of having duplicate declarations. See https://www.promptfoo.dev/docs/configuration/guide/#import-providers-from-separate-files.
d
Thank you so much for following up! I did some testing today and made progress, but I'm still not quite over the finish line. I think you're right that I need to read the provider from a file though as that got me past the error I was hitting before. But now I'm getting a 401 error when I try to use the attached provider1.yaml for the llm-rubric test, so I'm now I'm trying to debug that with the API developers. I put together provider2.yaml as an experiment and it doesn't throw any errors. I just get a message saying that the test failed - both the output and the rubric are empty because I can't get it to actually populate {{rubric}} and {{output}}. Is it actually possible to populate these? Or do you think this weird approach is a dead end? https://cdn.discordapp.com/attachments/1343743556505960459/1407105758029549579/promptfooconfig.yaml?ex=68a4e4db&is=68a3935b&hm=4db81b4c503845def968e3143dc4ea5e019827847d4e0dd13d17eb720f9acca2& https://cdn.discordapp.com/attachments/1343743556505960459/1407105758494986250/provider1.yaml?ex=68a4e4db&is=68a3935b&hm=e4edbb2e2b651e0e2ef69df1b6573c4ed61f1575557ffc1b518847f9ab2bc9d0& https://cdn.discordapp.com/attachments/1343743556505960459/1407105758943772724/provider2.yaml?ex=68a4e4db&is=68a3935b&hm=0b743e9cbd5051c55ce93f57a4c4349cb91d7849754ee018126043fbe34a6451& https://cdn.discordapp.com/attachments/1343743556505960459/1407105761620004975/fail.png?ex=68a4e4dc&is=68a3935c&hm=bef86234f3fad3d07d8fcb14f8421ec2e46c95fdef849961dd3e62e9bee51891&
w
Your code looks good but this is actually a bug on our end, @Daina – I've opened a PR to fix it, and this will likely be included in the next release!
Alternatively, you may be able to use the OpenAI provider and overwrite the
apiBaseUrl
parameter. Docs here: https://www.promptfoo.dev/docs/providers/openai/#configuring-parameters
d
Thanks again for following up! Unfortuantely overwriting the apiBaseUrl didn't alter anything. I think I'm going to have to recommend we use the default grader for now and see what the feedback is. I'll watch for updates in the next release though and keep working with the Credal API devs too
w
Try with
apiHost
instead
Something like
apiHost: https://app.credal.----.gov/api/openai
Either way, https://github.com/promptfoo/promptfoo/pull/5315 has been merged and will go out in the next release 😃