https://promptfoo.dev/ logo
How to set up similarity check via local Embedding...
# ❓|questions
a
I have downloaded the `BAAI/bge-large-en-v1.5 `model to conduct the similarity checks locally, without any APIs, etc. I attach the structure of the ./bge as the photo here. I am not sure how to properly write it down in the defaultTest:options, so I created a python script called
gte.py
to take care of the similarity scores. Again, I attach the pic to this post. In the promptfooconfig.yaml I wrote down this:
Copy code
defaultTest:
  options:
    provider:
      embedding:
        id: 'python:gte.py'
Which leads to it saying its not appropriate embedding provider and reverting to the OpenAI API:
Copy code
Provider python:gte.py is not a valid embedding provider for 'similarity check', falling back to default
Provider python:gte.py is not a valid embedding provider for 'similarity check', falling back to default
...
[FAIL] Error: OpenAI API key must be set for similarity comparison
Error: OpenAI API key must be set for similarity comparison
at OpenAiEmbeddingProvider.callEmbeddingApi (/opt/homebrew/lib/node_modules/promptfoo/dist/src/providers/openai.js:88:19)
I guess my setup is incorrect, but I am not sure where, as I tried to mimic the documentation. Any help would be appreciated. I used for reference: https://promptfoo.dev/docs/configuration/expected-outputs/similar/ https://promptfoo.dev/docs/providers/python https://cdn.discordapp.com/attachments/1247231122010542172/1247231122367316070/pic.png?ex=665f4600&is=665df480&hm=8a0c82ee409b76573a7c553b57346d2296f23c8b491fcc2793f3eef0ef659f33& https://cdn.discordapp.com/attachments/1247231122010542172/1247231122656591892/pic2.png?ex=665f4600&is=665df480&hm=e8c6d9b5d57caee2f7e07809ad054db42d12924ed671e5593622339981b53990&
i
I think your setup is exactly right, but unfortunately python embedding providers aren't supported right now. This PR adds support and will be in the next release: https://github.com/promptfoo/promptfoo/pull/864
this is included in 0.62.0
2 Views