https://promptfoo.dev/ logo
set llm rubric to azure [resolved but still flaggi...
# ❓|questions
a
i'm trying to do test assertions and am getting errors regarding using azure instead of the default openai model
Copy code
yaml
tests:
  - description: "does not describe self as an AI, model, or chatbot"
    assert:
      - type: llm-rubric
        value: does not describe self as an AI, model, or chatbot
        # need to figure out how to specify the azure provider here
        provider: azureopenai:chat:gptDeployment
error
Copy code
cmd
[FAIL] Error: Azure OpenAI API host must be set

Error: Azure OpenAI API host must be set
    at AzureOpenAiChatCompletionProvider.callApi (C:\Users\York Laptop 001\AppData\Roaming\npm\node_modules\promptfoo\dist\src\providers\azureopenai.js:174:19)
    at matchesLlmRubric (C:\Users\York Laptop 001\AppData\Roaming\npm\node_modules\promptfoo\dist\src\matchers.js:164:38)
    at async runAssertion (C:\Users\York Laptop 001\AppData\Roaming\npm\node_modules\promptfoo\dist\src\assertions.js:507:17)
    at async runAssertions (C:\Users\York Laptop 001\AppData\Roaming\npm\node_modules\promptfoo\dist\src\assertions.js:58:24)
    at async Evaluator.runEval (C:\Users\York Laptop 001\AppData\Roaming\npm\node_modules\promptfoo\dist\src\evaluator.js:208:37)
    at async C:\Users\York Laptop 001\AppData\Roaming\npm\node_modules\promptfoo\dist\src\evaluator.js:465:25
then
Copy code
yaml
tests:
  - description: "does not describe self as an AI, model, or chatbot"
    assert:
      - type: llm-rubric
        value: does not describe self as an AI, model, or chatbot
        # need to figure out how to specify the azure provider here
        provider:
          - id: azureopenai:chat:gptDeployment
            config:
              apiHost: "myendpoint.openai.azure.com"
gets error [next message]
Copy code
cmd
[FAIL] Error: Invariant failed: Provider supplied to assertion must have an id

Error: Invariant failed: Provider supplied to assertion must have an id
    at invariant (C:\Users\York Laptop 001\AppData\Roaming\npm\node_modules\promptfoo\node_modules\tiny-invariant\dist\tiny-invariant.cjs.js:14:11)
    at loadFromProviderOptions (C:\Users\York Laptop 001\AppData\Roaming\npm\node_modules\promptfoo\dist\src\matchers.js:39:34)
    at getGradingProvider (C:\Users\York Laptop 001\AppData\Roaming\npm\node_modules\promptfoo\dist\src\matchers.js:59:31)
    at getAndCheckProvider (C:\Users\York Laptop 001\AppData\Roaming\npm\node_modules\promptfoo\dist\src\matchers.js:68:31)
    at matchesLlmRubric (C:\Users\York Laptop 001\AppData\Roaming\npm\node_modules\promptfoo\dist\src\matchers.js:163:31)
    at runAssertion (C:\Users\York Laptop 001\AppData\Roaming\npm\node_modules\promptfoo\dist\src\assertions.js:507:55)
    at runAssertions (C:\Users\York Laptop 001\AppData\Roaming\npm\node_modules\promptfoo\dist\src\assertions.js:58:30)
    at Evaluator.runEval (C:\Users\York Laptop 001\AppData\Roaming\npm\node_modules\promptfoo\dist\src\evaluator.js:208:74)
    at async C:\Users\York Laptop 001\AppData\Roaming\npm\node_modules\promptfoo\dist\src\evaluator.js:465:25
the id is right there-- how do i set this properly?
huh, i fixed it by putting the details in the
defaultTest.options
Copy code
yaml
defaultTest:
  options:
    provider:
      id: azureopenai:chat:gptDeployment
      config:
        apiHost: "myendpoint.openai.azure.com"
tests:
  - description: "does not describe self as an AI, model, or chatbot"
    assert:
      - type: llm-rubric
        value: does not describe self as an AI, model, or chatbot
i
I think the problem was that you made
provider
an array in the original example, instead of an object. Before:
Copy code
yaml
tests:
  - description: "does not describe self as an AI, model, or chatbot"
    assert:
      - type: llm-rubric
        value: does not describe self as an AI, model, or chatbot
        # need to figure out how to specify the azure provider here
        provider:
          - id: azureopenai:chat:gptDeployment
            config:
              apiHost: "myendpoint.openai.azure.com"
Probable fix (remove
-
before provider id):
Copy code
yaml
tests:
  - description: "does not describe self as an AI, model, or chatbot"
    assert:
      - type: llm-rubric
        value: does not describe self as an AI, model, or chatbot
        # need to figure out how to specify the azure provider here
        provider:
          id: azureopenai:chat:gptDeployment
          config:
            apiHost: "myendpoint.openai.azure.com"
a
ah perhaps. let me see
i
I'll make a change to improve the error message in this case
a
you were right! my bad. not 100% confident with yaml in a conceptual sense
i
yaml is a pain sometimes 😦
a
thanks for the help
i
Pushed a small change that improves the error message and perhaps would have saved you time:
a
error messages are definitely a bear