https://promptfoo.dev/ logo
Compare output of different models
# ❓|questions
f
I am looking to evaluate the results of finetuning projects of different models. I would like to use assertions to verify that parts of the output of one model matches the outputs of another. I.e. that parts of the finetuned student model's output matches the teacher model. Is this possible?
i
I think technically this is possible using the [
select-best
](https://www.promptfoo.dev/docs/configuration/expected-outputs/model-graded/#examples-comparison) assertion, and then perhaps overriding the [rubric prompt](https://www.promptfoo.dev/docs/configuration/expected-outputs/model-graded/#select-best-rubric-prompt) to ask it to ensure that output B is similar to output A, but it's a bit of a hack/workaround
f
Interesting. I will give that a try.
a
Hi @User I wanted to do something similar, where I want to match output present in model to output in another model. In select-best, it is using LLM to select the best output based on the rubric prompt. But I want to do something deterministic. Say, if both models outputs list of 10 words, how many words are common in both the outputs. If I have more than 2 outputs. I want to show common words between all 3 models, common words between model 1&2, 2&3 and 1&3 etc.
@michaelmichaelmichael Can you help here. I wanted to know if we can do inter-model comparision in the same eval?
m
@abdksyed @felixthekraut This is where we run select best https://github.com/promptfoo/promptfoo/blob/d0fc8ec98227b66b6071e1b0352da007d9a52557/src/evaluator.ts#L782 We can definitely discuss adding a new deterministic metric but our capacity to work on it is limited. I would be happy to do a quick call with you if this is something you would like to work on in promptfoo https://cal.com/michael-dangelo/15min