https://promptfoo.dev/ logo
Testing Azure Web App
# ❓|questions
k
I've created a chatbot in portal.azure.com of type "Web App" that is configured to use a given
AZURE_SEARCH_INDEX
of a given
AZURE_SEARCH_SERVICE
. I would like to use Promptfoo to eval different `AZURE_OPENAI_MODEL`s with different `AZURE_OPENAI_SYSTEM_MESSAGE`s. I've looked at https://www.promptfoo.dev/docs/providers/azure/ but I'm unclear on how to accomplish this. Am I going about this wrong? Many thanks in advance!
I guess one option is to try to "recreate" my bot in Promptfoo and toy around with models and system prompts from there, but I still wouldn't know how to plug in my Azure search service and all the data it indexed.
u
@kentyman I can help you out here. I'm not super familiar with Azure, I have an account though. How/where specifically did you create the bot? If I'm on this page.. https://portal.azure.com/#allservices what am I looking for?
k
This was a collaborative effort so I didn't do all the steps. I have gotten a little farther though, and I can share that here:
I took a step back and took Promptfoo out of the equation to see if I could get anything working from Python. I found this tutorial that also has good explanations of how to get each necessary variable from the portal:
I got it running, but it still doesn't seem like it's accessing the data in my search index. Whenever I ask it a simple question it should know the answer to, it says:
assistant: The requested information is not available in the retrieved data. Please try another query or topic.
I do think I'm using the right search index because if I change it to gibberish, I get:
"No index with the name \'gibberish\' was found in the service \'chatbot-search-service1234512345\'."
Switching tactics, I went to the Chat playground in Azure AI Foundry, plugged in my index, and confirmed it could give me good answers using my index
Then, I clicked View code to see sample Python code that should work the same way. So right now I'm trying to see if I can glean anything from that code that will fix the previous code. The hope if getting this all working will illuminate which values I need to plug into Promptfoo.
Okay, made some progress. By comparing the queries in the above link and what I found in _Chat playground_'s View code, it looks like adding these to
parameters
got it to start answering questions:
Copy code
python
"embedding_dependency": {
    "type": "endpoint",
    "endpoint": "https://REDACTED.openai.azure.com/openai/deployments/text-embedding-ada-002/embeddings?api-version=2023-03-15-preview",
    "authentication": {
        "type": "api_key",
        "key": REDACTED,
    }
},
"query_type": "vector_simple_hybrid",
Ah, I think
query_type
defaults to
simple
, and that's too simple to be useful for my query
If I set it to
semantic
, it works even without the
embedding_dependency
Only if I use one of the three that start with
vector
do I need that embedding stuff: https://learn.microsoft.com/en-us/azure/ai-services/openai/reference#azuresearchquerytype
Whew. Okay, now that I can do it programmatically, I will try to get it working within Promptfoo...
Dang, I'm still getting:
[ERROR] API response error: 404 Resource not found
Could the problem be that I need
- type: azure_search
instead of
- type: AzureCognitiveSearch
? All of the docs seem to assume the latter
The docs say that
deployment_id
corresponds to "Azure cognitive services deployment ID.", but I don't really know what that is
Do they want the model (
gpt-4o
) already mentioned in
id: azure:chat:gpt-4o
? Do they want the embedding model (
text-embedding-ada-002
) for vector embeddings? Do they want the name of my Search service (
chatbot-search-service1234512345
) already mentioned in the
endpoint
? I'm not sure!
Sorry for the firehose. Kinda at a loss at this point. If anyone who's been successful with the Azure stuff can chime in, that would be great!
So it's not lost in the noise, this is a key question I have about the support
u
this is a lot of great information, I'll dig into it later tday
k
While not totally working, I have learned a few more things: 1. The 404 appears to be coming from my
dataSources
section, as if I comment it out I get a different error below... 2. Contrary to the documentation,
document_id
appears to no longer be the correct casing, as I get this error:
[ERROR] API response error: null Unrecognized request argument supplied: deployment_id
. Indeed, if I change it to
documentId
(but leave off
dataSources
), I get a response from the LLM (albeit with no knowledge of my data source).
So at the very least,
document_id
needs to change to
documentId
in the documentation. I can make that change and put up a PR.
But I'm still unsure why I can't add my data source, but I suspect it has something to do with the
type
being
azure_search
instead of
AzureCognitiveSearch
. Perhaps Promptfoo needs changes to support the former instead of just the latter?
So not sure why my promptfoo (0.103.19) says "Unrecognized request argument supplied: deployment_id"
Unless that message is from Azure and they broke compatibility?
Confirmed: " Formerly named Azure Cognitive Services, Azure AI services [...]": https://learn.microsoft.com/en-us/fabric/data-science/ai-services/ai-services-overview
Ah-ha! https://learn.microsoft.com/en-us/azure/ai-services/openai/references/on-your-data > Since API version
2024-02-15-preview
we introduced the following breaking changes comparing to earlier API versions: > - The API path is changed from
/extensions/chat/completions
to
/chat/completions
. > - The naming convention of property keys and enum values is changed from camel casing to snake casing. Example:
deploymentName
is changed to
deployment_name
. > - The data source type
AzureCognitiveSearch
is changed to
azure_search
. > - The citations and intent is moved from assistant message's context tool messages to assistant message's context root level with explicit schema defined.
It appears that
DEFAULT_AZURE_API_VERSION = '2024-12-01-preview'
, so I'm not sure why stuff didn't break.
v
Hey @kentyman, I was about to reply on Github but I figured it would just be easier to continue the conversation here. TL;DR, we were trying to make a change to switch our default GPT providers from OpenAI to Azure and the versions of some of the models we were using weren't available with the 2023 APIs. We've put that effort on hold, I can revert for now and when we re-land this change we can look at what the best option might be but that could also be implementing a provider that has support for older Azure API versions.
k
Thanks for the quick response! I don't want to undo any progress you made. I'm more just trying to tease apart whether anything needs updating since that API version changed (and then hopefully that fixes my original issues)
v
All good! Sorry if you had to do some protracted debugging to figure out why things were breaking. I'll just revert the change for now and I'll reach back out when we change this
k
No worries. Though while I have your ear, do you have any insight into why: 1. I have to use
deploymentId
instead of
deployment_id
like
azure.ts
seems to reference? The answer to that may help me decide whether I need to update the documentation. 2. I get 404 when I add
dataSources
.
For that #2 data source issues, maybe it would be helpful to restate exactly what I'm trying:
Copy code
yaml
dataSources:
  - type: azure_search # AzureCognitiveSearch doesn't work either!
    parameters:
      endpoint: 'https://MY-SEARCH-SERVICE.search.windows.net/'
      key: 'REDACTED'
      indexName: 'MY-INDEX-NAME' # index name under MY-SEARCH-SERVICE
Please let me know if I'm grabbing from the wrong places
v
I'll take a deeper look at this later today if that works? I have some context on the Azure provider. I'm not an expert on the Azure OAI + PF integration and so I'll just need to familiarize myself
k
As a workaround, I made a
provider.py
that does work. Sharing here (with certain values redacted with
MY-FOO
variants) in case it helps shed light on why the YAML isn't working:
@Vedant did you happen to get a chance to look at this?
v
Hey @kentyman , sorry for the delay! We had a few things that came up last week but I will look into this today.
k
No worries and many thanks!
Checking in on this, but not definitely not asking you to spend your weekend on it. 🙂
Friendly reminder about this. Thanks!
Any chance you can look at this sometime soon?
m
Hey @kentyman, thanks again for your patience, and I'm sorry about the delay here. This is on our backlog and your detailed breakdown is extremely helpful. If you’re interested and have time, we would love help fixing it in the promptfoo codebase, or we could pair together and as a thank you, I’d be happy to send you some Promptfoo swag. Really appreciate your thorough debugging and persistence. Let me know if you’re up for it.
k
No worries, I understand y'all are busy. I'm happy to try to fix it (with help), though I'm also busy so we'll have to work around existing priorities. 🙂 But I kinda hit a wall with the above, so definitely need help with next steps.
@michaelmichaelmichael I'm looking into this again. Any thoughts on what to try next?
Doesn't look like @fluorescentpangolin2491 is here anymore
Checking in again
m
I've got some decent experience with using AOAI as well as Azure AI Search as a vectorstore. i can try to take a look and see if I can resolve the issues you are facing 🙂
k
That would be awesome! I worry that Promptfoo may have some bugs or other issues preventing easy integration. People can work around it with a
provider.py
, but that's obviously not as ideal. Let me know if you have anything I should try. Please see all of this thread for context on exactly what I was doing and the errors I was getting. Thanks again!
2 Views