https://promptfoo.dev/ logo
azureopenai and azure api host
# ❓|questions
r
Hello! This library looks awesome and I'm really excited to try it out. I'm trying to use it with azureopenai, and I'm getting this error: [FAIL] (0.00) Error: Azure OpenAI API host must be set Is there a way to set the AZURE_OPENAI_API_HOST in promptfooconfig.yaml, or does this need to be set in my environment variables? When I try adding it to the promptfooconfig.yaml like this: prompts: [prompts.txt] providers: id: azureopenai:chat:gpt-4-32k config: apiHost: 'https://xxxxxxx.openai.azure.com/' I get the same error. Thanks!
i
hi @roneyal, You should be able to set the host either by
AZURE_OPENAI_API_HOST
environment variable or under the
env
key in your config:
Copy code
yaml
env:
  AZURE_OPENAI_API_HOST: "https://xxxxx.openai.azure.com"
Let me know if you still run into issues! The
apiHost
key is currently not supported by the Azure provider (though I've just committed a [fix](https://github.com/promptfoo/promptfoo/commit/c7c0949f1a6237200834236247cef586f510de21) that enables this)
r
Thanks, @ian for the quick reply and for updating the code! I tried that first option, adding this to my config.yaml:
Copy code
env:
  AZURE_OPENAI_API_HOST: "https://xxxxxxxx.openai.azure.com"
But I still got the same error (Error: Azure OpenAI API host must be set) When I try this:
Copy code
providers:
  - id: azureopenai:chat:gpt-4-32k
    config:
      apiHost: "xxxxxxxx.openai.azure.com"
I get a different error:
Copy code
[FAIL] (0.00) API response error: TypeError: Cannot read properties of undefined (reading '0'): {""error"":{""code"":""DeploymentNotFound"",""message"":""The API deployment for this resource does not exist. If you created the deployment within the last 5 minutes, please wait a moment and try again.""}}
---
Do you happen to know how to resolve that? -- Update - I resolved this error. I figured out that my custom Azure deployment, which I believe is also called the engine, should be used like this:
Copy code
providers:
  - id: azureopenai:chat:engineNameGoesHere
    config:
      apiHost: "xxxxxxxx.openai.azure.com"
When I try to run
Copy code
npx promptfoo view
it says:
Copy code
Server listening at http://localhost:15500
Do you want to open the browser to the URL? (y/N):
When I open it in a browser it just says:
Copy code
Cannot GET /
Any idea why that might be? Thanks!
j
@roneyal are you locally `npm install`ing? I get that "Cannot GET /" error when locally `npm install`ing via
npm install ../promptfoo
i
that's a bit strange... what about http://localhost:15500/eval ? can you also make sure you're running the latest:
npx promptfoo@latest view
r
Hi @jamesbraza and @ian Thanks for your quick replies and sorry for the delay! @jamesbraza - Yes I think I initially installed it with
Copy code
npm install promptfoo
Is that recommended? I think I did that because the npx commands weren't working for me until I did that. @ian Yes even http://localhost:15500/eval doesn't work. However, when I try
Copy code
npx promptfoo@latest view
, that does work, so that's good! Thanks for that suggestion. I assume that means when I'm running
Copy code
npx promptfoo view
, I'm not using the latest version, though I'm not sure why that is happening because I've pulled the latest version.
I have another question -- I'm trying to do an assert with python like this:
Copy code
assert:
  - type: python
  value: len(output.split())
And I'm getting this error:
Copy code
Python code execution failed: spawnSync /bin/sh EPIPE
Any idea of how to resolve that? Thanks!
i
assuming that's not a copy/paste error, it may be an indentation issue:
Copy code
yaml
assert:
  - type: python
    value: len(output.split())
If not that, then it looks like it's trying to run a
python
executable that doesn't exist (or it doesn't have permissions to execute it). Do you have a
python
executable in the context that you're running promptfoo? As a workaround, you can try the equivalent Javascript:
Copy code
yaml
assert:
  - type: javascript
    value: output.split(" ").length;
glad you got it working with
npx promptfoo@latest
I'm curious what your installed version of promptfoo is globally. Can you run
npm list -g promptfoo
?
r
Thanks Ian! I realized that the python executable wasn't available in the context I was running promptfoo, and resolving that fixed it. (The Javascript alternative also worked fine).
@ian When I run
Copy code
npm list -g promptfoo
I get:
Copy code
% npm list -g promptfoo
/opt/homebrew/lib
└── (empty)
Does that mean I haven't installed it properly, or at all? I've been running promptfoo with
Copy code
npx promptfoo eval -o output.csv
i
ah ok, so there's no global installation. What about
npm list promptfoo
You can also try reinstalling with
npm install promptfoo@latest
and see if the problem is fixed. If you're happy with using
npx
, feel free to ignore 🙂
r
Copy code
npm list promptfoo
gives:
Copy code
% npm list promptfoo
promptfoo@0.24.4 /Volumes/Drive/promptfoo/promptfoo
└── (empty)
After reinstalling with
Copy code
npm install promptfoo@latest
I get:
Copy code
% npm list promptfoo          
promptfoo@0.24.4 /Volumes/Drive/promptfoo/promptfoo
└── promptfoo@0.25.0
After doing that, I still get the same error as above when I try to run
Copy code
npx promptfoo view
Copy code
npx promptfoo@latest view
still works though! Is there a way to run promptfoo view with npm, now that I've installed it?
3 Views