https://promptfoo.dev/ logo
Connection to azure openAI,
# ❓|questions
k
so I am using node to make the connection to azure openAI, using the following code: const providers = [{ id: 'azureopenai:chat:gpt-testing', config: { apiHost: "***.openai.azure.com", apiKey: "***", } }]; And it's working perfectly. But in azure openAI, there is the the Add your data source where i created mine, and inserted a couple pdf files. in order to access that i need to make additional connections to azure cognitive search. and i was wondering how can i add those in my providers variable.
@ian is it possible you could help? it's kinda urgent
i
Do you know what the actual chat completion endpoint is for your model?
or, if the problem is that you want to run a search before calling your model, you'd have to write a custom provider https://promptfoo.dev/docs/guides/evaluate-rag
k
@ian the problem is that i can't find the proper provider config attributes In the prompt foo NPM package, so i can apply the config Azure is suggesting using python. I need to apply it in the promptfoo nodejs library. I tried to do what seen in the picture but it doesn't seem to work
i
I see. The
dataSources
key is not supported e.g. and
https://github.com/promptfoo/promptfoo/commit/3f6dee99b4ef860af1088c4ceda1a74726070f37 adds support for
deployment_id
and
dataSources
in the configs field. An example value of
dataSources
is:
Copy code
js
dataSources=[
    {
        "type": "AzureCognitiveSearch",
        "parameters": {
            "endpoint": "...",
            "key": "...",
            "indexName": "...",
        }
    }
]
or yaml equivalent 🙂
k
Thanks for your feedback @ian , I am getting this error : API response error: TypeError: Cannot read properties of undefined (reading '0'): {\"error\":{\"message\":\"Unrecognized request argument supplied: dataSources\",\"type\":\"invalid_request_error\",\"param\":null,\"code\":null}} Saying that the datasource is not a valid request argument ?
i
can you share your config?
k
this my code @ian
hey @ian i sent another PR https://github.com/promptfoo/promptfoo/pull/274 to parse the messages according to the response recieved from AzureOpenAI
i
thanks 🙏
will publish a release with these fixes soon
k
hello @ian at the moment i have added a second provider for embedding to be able to work with the similarity assertion type, but I am receiving this error in the embedding provider: {provider: {id: "azureopenai:embedding:text-embedding-ada-002"},…} error : "Error: Not implemented\n\nError: Not implemented at AzureOpenAiEmbeddingProvider.callApi
i
The main
providers
list is for prompting. To set this as the embedding provider, you must set it on the test case itself or in
defaultTest
. See the example here 🙂 https://promptfoo.dev/docs/configuration/expected-outputs/similar#overriding-the-provider
k
thank you for clarifying, it's working great now!
hello again @ian , i know what i put in my prompt is not good, but i was wondering how can i fix it.
i
hmm it looks like you have a typo,
conetent
?
k
yeah i fixed it but it still doesnt work, its only working right now if i do this :
and i was wondering if i can seperate the behavior from the prompt
i
I see. are you getting an error?
two more things: 1.
prompts
should be an array of prompts, not a single prompt array (you're missing wrapping
[ ]
). here's an example https://github.com/promptfoo/promptfoo/blob/main/examples/node-package/index.js#L6-L19 2. You'll have to update to version
0.27.0
- I've added support for prompt objects in calls to
evaluate
k
hello, thanks for the help, i made a pull request to adapt to the new changes
i
The 0.27.0 changes should be good to go with the above example. We still want to keep the ability to eval an array of separate prompts 🙂
k
i already tested with 0.27.0 and it did not work
thats why i made the change
i
can you show me your config?
and elaborate on "did not work"
k
Well my config is as following: prompts: [ { role: "system", content: "You are a ecommerce salesperson who cant wait to help me to know more about Shopify. You will talk to me in a very bubbly and enthutiast manner" }, { role: "user", content: data[i].questions } ], but when azure api was being called at the end it was called only with : https://test-solution.openai.azure.com/openai/deployments/gpt-testing/extensions/chat/completions?api-version=2023-07-01-preview "messages": [ { "role": "system", "content": "You are an ecommerce chat assistant for shopify. Answer the user's question by starting with: Dear customer,|" }, ] and this returns the following error : "error": { "requestid": "be778340-681e-43c1-9ab3-28ae1afb353a", "code": 400, "message": "Invalid message request body. Learn how to use Completions extension API, please refer to https://learn.microsoft.com/azure/ai-services/openai/reference#completions-extensions" } it is done like that because the code initially iterates on the array of strings and sent an api call per prompt. whereas here the prompt is OpenAI Message and should be sent in one API Request as: "messages": [ { "role": "system", "content": "You are an ecommerce chat assistant for shopify. Answer the user's question by starting with: Dear customer,|" }, { "role": "user", "content": "how much is shopify" } ],
My PR make the differentiation between prompts which are array of string (raw messages) and prompts that are OpenAI object messages
i
it looks like you are passing a single prompt to
prompts
, but it expects an array of prompts. You should wrap it like so:
Copy code
js
prompts: [
  [
    {
      role: 'system',
      content: '...',
    },
    {
      role: 'user',
      content: '...',
    },
  ]
],
k
ah an array of arrays ?
humm that would work ?
i
it should 🙂
k
let me try and get back to you then
yup that makes my PR useless. thanks !
i
ok! let me know if you run into any other problems. I appreciate you raising these issues
k
Yep no worries, thank you for all your effort and availability too, appreciated!
I am debugging now the similarity part of the checkof asserrtion because the scores are a little bit weird to me let me show you two examples if i may :
0.73 seems a high score for similarity between potato and 23 .. also between a response that says left and right hand (complete opposite answer) to have 0.9+ is too much of a score no ?
I expect below 0.5 scores for both these answers the least
i
unfortunately that's just what OpenAI Ada embeddings return. They are not perfect, especially for small text I responded to a similar issue here where I verified the scores are being directly returned from OpenAI: If desired, you could select alternative text embeddings from HuggingFace
k
it has a better similarity check ?
yea i could try that
13 Views