https://promptfoo.dev/ logo
similarity testing inconsistent
# ❓|questions
k
Hello, I am getting a very high similarity score when the result is just wrong. prompt : get information on employee number 001. expected : you want information on employee number 002, is that it? response from azure open ai : you want information on employee number 001, is that correct? and i get a similarity score of 0.83 . meanwhile when the expected is : you want information on employee number 001, is that it? i get a similarity score of 0.84 . Is there something i can do to make the scores more accurate ?
i
similarity is probably the wrong test here. The output vs expected is very similar (similarity score just uses OpenAI embeddings) I would use
llm-rubric
or a deterministic test such as
contains
instead 🙂
k
thank you, model-graded-closedqa was the best choice in my case