[Promptfoo] Konstantine
03/09/2026, 11:32 PMtests:
- vars:
input: "Hello"
assert:
- type: latency
threshold: 5000 # max milliseconds allowed
This will pass/fail based on whether the response time is under the threshold. Make sure to use --no-cache since latency isn't tracked for cached results.
For token usage, you can use a JavaScript assertion:
assert:
- type: javascript
value: "context.providerResponse.tokenUsage.total < 1000"
There's also a built-in cost assertion if you want to cap your spend per call:
assert:
- type: cost
threshold: 0.01 # max USD
metric: cost_usd
Regarding the UI metrics dropdown, that filters by custom named scores, not latency/tokens directly. Latency and token usage are shown in the summary header per provider column. If you want latency to appear as a filterable metric in the dropdown, add the metric field:
assert:
- type: latency
threshold: 5000
metric: latency_ms
Docs:
https://www.promptfoo.dev/docs/configuration/expected-outputs/#deterministic-eval-metrics