https://promptfoo.dev/ logo
Cost/ spendings:
# ❓|questions
g
How many tokens are your evaluations using? My current set up has 50 test questions with ~6 evaluations for each (all model-graded). Running this setup i have approximated takes around 300k tokens which approximates to 10$ per run. I like the large test set data set to create data sufficiency. Note: I am loading the context dynamically as seen in the below figure. https://cdn.discordapp.com/attachments/1225412491975069706/1225412493044748388/image.png?ex=66210951&is=660e9451&hm=5bcd4c031b6826034a682773c68bd23526e56a3c875bfbdf662af9d3edfc4422&
i
The token usage of the evals themselves is fairly low. The largest prompt (
factuality
) is 200 tokens.
llm-rubric
is 140. The vast majority of token usage can be attributed to the inputs/outputs themselves