https://promptfoo.dev/ logo
Inconsistent model graded metrics
# ❓|questions
m
I've tried gpt-4o, claude-3.5-sonnet, and gemini-2-flash. I've made sure I set temperature to 0 and fixed the seed.
Copy code
defaultTest:
  options:
    provider: vertex:gemini-2.0-flash-001
    temperature: 0
    seed: 248079012
I've made absolutely sure that the input text being fed to the evals is the same every time. I've tried llm-rubric, g-eval, and model-graded-closed-qa. And yet.. I get inconsistent evaluations. Sometimes I'll get 8 pass 2 fail, sometimes 6 pass and 4 fail, etc. I don't see how I can compare old to new prompts if I can get consistent results with the same prompt. Here's an example of two different outputs from llm-rubric for
Praised salesperson for active listening techniques - 'While explaining her questions and buying decision you continually signaled you were actively listening with affirmations'
Fail:
The content praises the salesperson's listening skills, stating: 'You excelled at being a good listener and encouraging Alisa to express her thoughts. Throughout the call, you allowed her to share her concerns and considerations extensively. This approach builds trust and helps you understand the customer's needs better.' However, it doesn't mention anything about the use of affirmations specifically.
Pass:
The review praises the salesperson's listening skills by saying, 'You excelled at being a good listener and encouraging Alisa to express her thoughts. Throughout the call, you allowed her to share her concerns and considerations extensively.' While it doesn't specifically mention 'affirmations,' it does highlight the salesperson's ability to let the customer express herself, indicating active listening.
I wonder if I'm just missing something obvious, because this seems like it would be a showstopper for everyone...
My conjecture based on some additional testing is that temperature is getting ignored in
Copy code
defaultTest:
  options:
    provider: openai:gpt-4o
    temperature: 0
(I say this because I did another test where I replaced the rubric prompt to choose a random word, and I get all completely different words. I know the LLM can be non-deterministic even with temp 0, but pretty sure not this random.)
i
hi michael, interesting theory. you may be able to verify this by looking at the output with the
--verbose
flag appended to your eval command
in the meantime, I will investigate
m
Good call, @ian ! I had tried that with my full project and it was truncating the output too much, but with my toy project, here's the smoking gun:
Calling OpenAI API: {"model":"gpt-4o","messages":[{"role":"user","content":"Say 'hello' and nothing else"}],"max_tokens":1024}
No temperature in the call
Here's my YAML
Copy code
description: "Temperature test"

providers:
  - openai:gpt-4o

prompts: 
  - "Say 'hello' and nothing else"

evaluateOptions:
  maxConcurrency: 10
  repeat: 10

defaultTest:
  options:
    provider: openai:gpt-4o
    temperature: 0
    seed: 248079012
    rubricPrompt: file://g2.yaml
    
tests:
  - assert:
      - type: llm-rubric
        value: Ignore me
        options:
          provider: openai:gpt-4o
          temperature: 0
          seed: 248079012
Oh, so I could probably workaround with a custom provider for the assert?
i
ah I believe you're nesting the properties incorrectly. Can you try something like this?
Copy code
yaml
defaultTest:
  options:
    provider:
      id: openai:gpt-4o
      config:
        temperature: 0
        seed: 1234
m
Oh, derp! That helped quite a bit, now I got 8 apples and 2 bananas - so, not perfectly consistent but waaaay better than before. Thank you!
i
noo problem I wish the nested properties were a bit less convoluted 🥲