https://promptfoo.dev/ logo
Extracting Metrics
# ❓|questions
s
I'm running a test which introduces 2 personas, and calculates a "Suitability" score for a supplied article. I'm using a JavaScript assert to capture the score for each persona, and then subtracting one from the other to calculate the different. I'm exporting this as a metric "score-diff". I am running against a few LLMs and using --repeat to see how much variability there is in the score calculation. Is there an easier format to get those scores in a plottable format than parsing the JSON output file?
i
hmm, unfortunately not right now. we don't have first class support for looking at variability of repeat requests, although that is something I would like to add
s
thank you for taking the time to respond ian -- i was being a lazy 🙂 i'm doing quite a bit of this type of research right now so happy to let you know how i get on if that's useful.
i
you're not the first person to ask! let me know if it winds up being a blocker for your work
s
not at all, sonnet 3.5 spat out a python script that extracts the data perfectly, so i'm all set. ALSO -- just to double check (having run a few tests) -- it looks like the Anthropic provider uses a TEMPERATURE of 0 if none is specified (i believe 1 is the default in the anthropic API docs). I've re-run with the environment variable set and getting different results - but good to confirm.
( anthropic.ts line 212 ) - pretty sure i've read that right and can't see any other overrides.
similar thing with openai.ts line 287 - would be nice to know for sure if default is 0 unless specified.
(sorry, ignore the above i've just seen in the documentation the default is 0, somehow missed that)
OK - there seems to be an issue with temperature setting with the ANTHROPIC_TEMPERATURE environment variable.
this is the line from the promptfooconfig.yaml file:
providers: - "anthropic:messages:claude-3-sonnet-20240229"
❯ gci env:ANTHROPIC_T* Name Value ---- ----- ANTHROPIC_TEMPERATURE 0.8
❯ promptfoo cache clear && promptfoo eval --verbose Clearing cache... Reading prompts from ["simple_score.json"] ... ... E>"}]}],"stream":false,"temperature":0}
change promptfooconfig like so: providers: - id: "anthropic:messages:claude-3-sonnet-20240229" config: temperature: 1.0
and...
}],"stream":false,"temperature":1}
it seems to work OK for the OPENAI_TEMPERATURE environment variable, and ANTHROPIC_API_KEY must be working...
having a quick look at the code (i may get a chance to try a patch tomorrow) i reckon it's in the anthropic.ts MessageCreateParams. doesn't seem to get from process.env as CompletionCreateParams does.
@ian if you see this let me know if i'm on the right path
i
oh man you're right. here's a fix https://github.com/promptfoo/promptfoo/pull/997
s
thank you -- was about to try fixing myself!
2 Views