kentyman
02/03/2025, 11:15 PMAZURE_SEARCH_INDEX of a given AZURE_SEARCH_SERVICE. I would like to use Promptfoo to eval different `AZURE_OPENAI_MODEL`s with different `AZURE_OPENAI_SYSTEM_MESSAGE`s. I've looked at https://www.promptfoo.dev/docs/providers/azure/ but I'm unclear on how to accomplish this. Am I going about this wrong? Many thanks in advance!kentyman
02/04/2025, 3:17 PM[Promptfoo] Steve
02/04/2025, 5:58 PMkentyman
02/04/2025, 6:45 PMkentyman
02/04/2025, 6:47 PMkentyman
02/04/2025, 6:48 PMkentyman
02/04/2025, 6:48 PMassistant: The requested information is not available in the retrieved data. Please try another query or topic.kentyman
02/04/2025, 6:49 PMkentyman
02/04/2025, 6:50 PM"No index with the name \'gibberish\' was found in the service \'chatbot-search-service1234512345\'."kentyman
02/04/2025, 6:51 PMkentyman
02/04/2025, 6:51 PMkentyman
02/04/2025, 7:25 PMparameters got it to start answering questions:kentyman
02/04/2025, 7:26 PMpython
"embedding_dependency": {
"type": "endpoint",
"endpoint": "https://REDACTED.openai.azure.com/openai/deployments/text-embedding-ada-002/embeddings?api-version=2023-03-15-preview",
"authentication": {
"type": "api_key",
"key": REDACTED,
}
},
"query_type": "vector_simple_hybrid",kentyman
02/04/2025, 7:29 PMquery_type defaults to simple, and that's too simple to be useful for my querykentyman
02/04/2025, 7:30 PMsemantic, it works even without the embedding_dependencykentyman
02/04/2025, 7:31 PMvector do I need that embedding stuff: https://learn.microsoft.com/en-us/azure/ai-services/openai/reference#azuresearchquerytypekentyman
02/04/2025, 7:31 PMkentyman
02/04/2025, 7:58 PM[ERROR] API response error: 404 Resource not foundkentyman
02/04/2025, 7:59 PM- type: azure_search instead of - type: AzureCognitiveSearch? All of the docs seem to assume the latterkentyman
02/04/2025, 8:01 PMdeployment_id corresponds to "Azure cognitive services deployment ID.", but I don't really know what that iskentyman
02/04/2025, 8:04 PMgpt-4o) already mentioned in id: azure:chat:gpt-4o? Do they want the embedding model (text-embedding-ada-002) for vector embeddings? Do they want the name of my Search service (chatbot-search-service1234512345) already mentioned in the endpoint? I'm not sure!kentyman
02/04/2025, 8:10 PMkentyman
02/05/2025, 12:30 AM[Promptfoo] Steve
02/05/2025, 5:32 PMkentyman
02/05/2025, 7:05 PMdataSources section, as if I comment it out I get a different error below...
2. Contrary to the documentation, document_id appears to no longer be the correct casing, as I get this error: [ERROR] API response error: null Unrecognized request argument supplied: deployment_id. Indeed, if I change it to documentId (but leave off dataSources), I get a response from the LLM (albeit with no knowledge of my data source).kentyman
02/05/2025, 7:06 PMdocument_id needs to change to documentId in the documentation. I can make that change and put up a PR.kentyman
02/05/2025, 7:07 PMtype being azure_search instead of AzureCognitiveSearch. Perhaps Promptfoo needs changes to support the former instead of just the latter?kentyman
02/05/2025, 7:26 PMdeploymentId works! Sure looks like azure.ts only cares about `deployment_id`:
https://cdn.discordapp.com/attachments/1336112881007071232/1336779923259654215/image.png?ex=67a50cd1&is=67a3bb51&hm=71eb992071b3fa2c3bdb45f7b86d77ae971ac2b5a30073a49182dee704758a17&kentyman
02/05/2025, 7:30 PMkentyman
02/05/2025, 7:30 PMkentyman
02/05/2025, 7:35 PMkentyman
02/05/2025, 7:40 PMkentyman
02/05/2025, 7:47 PM2024-02-15-preview we introduced the following breaking changes comparing to earlier API versions:
> - The API path is changed from /extensions/chat/completions to /chat/completions.
> - The naming convention of property keys and enum values is changed from camel casing to snake casing. Example: deploymentName is changed to deployment_name.
> - The data source type AzureCognitiveSearch is changed to azure_search.
> - The citations and intent is moved from assistant message's context tool messages to assistant message's context root level with explicit schema defined.kentyman
02/05/2025, 8:03 PMDEFAULT_AZURE_API_VERSION = '2024-12-01-preview', so I'm not sure why stuff didn't break.kentyman
02/05/2025, 8:07 PMVedant
02/05/2025, 8:28 PMVedant
02/05/2025, 8:29 PM2024-12-01-preview is a supported Azure API version (wasn't a typo)
https://learn.microsoft.com/en-us/azure/ai-services/openai/api-version-deprecation#changes-between-2024-12-01-preview-and-2024-10-01-previewkentyman
02/05/2025, 8:30 PMVedant
02/05/2025, 8:32 PMkentyman
02/05/2025, 8:45 PMdeploymentId instead of deployment_id like azure.ts seems to reference? The answer to that may help me decide whether I need to update the documentation.
2. I get 404 when I add dataSources.kentyman
02/05/2025, 9:15 PMyaml
dataSources:
- type: azure_search # AzureCognitiveSearch doesn't work either!
parameters:
endpoint: 'https://MY-SEARCH-SERVICE.search.windows.net/'
key: 'REDACTED'
indexName: 'MY-INDEX-NAME' # index name under MY-SEARCH-SERVICEkentyman
02/05/2025, 9:20 PMendpoint (top-right field under Url):
https://cdn.discordapp.com/attachments/1336112881007071232/1336808736936689694/image.png?ex=67a527a7&is=67a3d627&hm=626b4a2fe29cbdaadaeec84071604f5e674846fb0a9cd4e9d6968cd4d41a510f&kentyman
02/05/2025, 9:22 PMkentyman
02/05/2025, 9:25 PMkentyman
02/05/2025, 9:25 PMVedant
02/05/2025, 10:22 PMkentyman
02/06/2025, 8:02 PMprovider.py that does work. Sharing here (with certain values redacted with MY-FOO variants) in case it helps shed light on why the YAML isn't working:kentyman
02/13/2025, 12:31 AMVedant
02/13/2025, 5:30 PMkentyman
02/13/2025, 5:38 PMkentyman
03/08/2025, 6:16 PMkentyman
03/13/2025, 12:20 PMkentyman
04/08/2025, 1:45 PMmichaelmichaelmichael
04/09/2025, 6:02 AMkentyman
04/09/2025, 3:19 PMkentyman
06/02/2025, 2:26 PMkentyman
06/04/2025, 12:57 PMkentyman
06/16/2025, 12:08 PMmjt2007
06/20/2025, 11:10 PMkentyman
06/22/2025, 2:53 PMprovider.py, but that's obviously not as ideal. Let me know if you have anything I should try. Please see all of this thread for context on exactly what I was doing and the errors I was getting. Thanks again!