Hi all, I'm trying to set up llm-rubric tests against a custom server using a config like:

providers:
  - id: openai:gpt-4.1-mini
    label: openai
    config:
      apiHost: "yourAIhost.com"
      apiKey: sk-abc123

So the prompts themselves are executed properly against "yourAIhost.com" and work properly. But the llm-rubric tests are then executed against

api.openai.com

I tried many different setups. Just the one I provided. I tried setting

defaultTest:
  options:
    provider: openai:gpt-4.1-mini

I tried using a custom label. I tried making the whole provider custom http provider, which also worked, and then to reference it in defaultTest, or in the llm-rubric test. Nothing works – the test always hits the api.openai.com When I tried to use custom https provider, with a label, and then reference it in defaultTest config, I get a response in the UI:

Error: Invariant failed: Expected HTTP provider https:custom to have a config containing {body}, but instead got {}

That looks like a bug, cause it identifies the provider, but not its config. Which works correctly for the prompts themselves, just not in llm-rubric verifier. Anyone had similar problem and managed to overcome it?

Can you share your entire config?

a minimal config to reproduce:

description: "My eval"

prompts:
  - "Write a tweet about {{topic}}"
  - "Write a concise, funny tweet about {{topic}}"

providers:
  - id: openai:jan-nano-4b-Q4_K_M
    label: openai
    config:
      apiBaseUrl: "http://localhost:1337/v1"
      apiKey: asdf1234
  

tests:
  - vars:
      topic: bananas

  - vars:
      topic: new york city
    assert:
      - type: llm-rubric
        value: ensure that the output is funny

running then with

promptfoo eval -v

The tweets are generated properly, but for "new york city" llm-rubric eval I see: Error: API key is not set. Set the OPENAI_API_KEY environment variable or add

apiKey

to the provider config. If I do provide the OPENAI_API_KEY, I get :

{
  "error": {
    "message": "Incorrect API key provided: asdf1234. You can find your API key at https://platform.openai.com/account/api-keys.",
    "type": "invalid_request_error",
    "param": null,
    "code": "invalid_api_key"
  }
}

and in the logs I see:

[defaults.js:160] Using OpenAI default providers
[chat.js:264] Calling https://api.openai.com/v1 API: {"model":"gpt-4.1-2025-04-14","messages":....

So it calls the main api.openai.com URL instead of the only defined provider's URL. The same happens when I set defaultTest: options: provider: openai:jan-nano-4b-Q4_K_M

You were correct in using

defaultTest.options.provider

to set the provider that's used for grading llm-rubric assertions (i.e. the "Grading Provider"). Provider configurations are not shared, by default, between the

providers

block and

defaultTest.options.provider

, so you would also need to re-set the

config.apiBaseUrl

on the Grading Provider i.e.

defaultTest:
  options:
    provider:
      id: openai:jan-nano-4b-Q4_K_M
      config:
        apiBaseUrl: "http://localhost:1337/v1"

I'm not entirely following the configurations that you mentioned in your original message having already tried; have you attempted to set the Grading Provider as such and, if so, what was that outcome?