michaelnglowforge
02/16/2025, 6:07 PMdefaultTest:
options:
provider: vertex:gemini-2.0-flash-001
temperature: 0
seed: 248079012
I've made absolutely sure that the input text being fed to the evals is the same every time.
I've tried llm-rubric, g-eval, and model-graded-closed-qa.
And yet.. I get inconsistent evaluations. Sometimes I'll get 8 pass 2 fail, sometimes 6 pass and 4 fail, etc. I don't see how I can compare old to new prompts if I can get consistent results with the same prompt.
Here's an example of two different outputs from llm-rubric for Praised salesperson for active listening techniques - 'While explaining her questions and buying decision you continually signaled you were actively listening with affirmations'
Fail:
The content praises the salesperson's listening skills, stating: 'You excelled at being a good listener and encouraging Alisa to express her thoughts. Throughout the call, you allowed her to share her concerns and considerations extensively. This approach builds trust and helps you understand the customer's needs better.' However, it doesn't mention anything about the use of affirmations specifically.
Pass:
The review praises the salesperson's listening skills by saying, 'You excelled at being a good listener and encouraging Alisa to express her thoughts. Throughout the call, you allowed her to share her concerns and considerations extensively.' While it doesn't specifically mention 'affirmations,' it does highlight the salesperson's ability to let the customer express herself, indicating active listening.
I wonder if I'm just missing something obvious, because this seems like it would be a showstopper for everyone...michaelnglowforge
02/17/2025, 3:16 AMdefaultTest:
options:
provider: openai:gpt-4o
temperature: 0michaelnglowforge
02/17/2025, 4:34 AMian
02/17/2025, 2:37 PM--verbose flag appended to your eval commandian
02/17/2025, 2:37 PMmichaelnglowforge
02/17/2025, 7:31 PMmichaelnglowforge
02/17/2025, 7:31 PMCalling OpenAI API: {"model":"gpt-4o","messages":[{"role":"user","content":"Say 'hello' and nothing else"}],"max_tokens":1024}michaelnglowforge
02/17/2025, 7:31 PMmichaelnglowforge
02/17/2025, 7:31 PMdescription: "Temperature test"
providers:
- openai:gpt-4o
prompts:
- "Say 'hello' and nothing else"
evaluateOptions:
maxConcurrency: 10
repeat: 10
defaultTest:
options:
provider: openai:gpt-4o
temperature: 0
seed: 248079012
rubricPrompt: file://g2.yaml
tests:
- assert:
- type: llm-rubric
value: Ignore me
options:
provider: openai:gpt-4o
temperature: 0
seed: 248079012michaelnglowforge
02/17/2025, 7:33 PMian
02/17/2025, 9:20 PMyaml
defaultTest:
options:
provider:
id: openai:gpt-4o
config:
temperature: 0
seed: 1234michaelnglowforge
02/17/2025, 10:03 PMian
02/18/2025, 12:36 AM