u1f40e
12/11/2024, 7:20 PM--repeat. Currently beforeEach runs once per input, rather than 5 times per input with --repeat 5.
Alternately, is there any unique per-test identifier we can access from the UI? I'm trying to push a unique id through our provider so we can search for traces specific to each test.michaelmichaelmichael
12/12/2024, 8:53 PMdescription: "Before Each Repeat Test"
prompts:
- "Write a tweet about: {{topic}}."
extensions:
- file://hooks.js:extensionHook
providers:
- openai:chat:gpt-4o-mini
tests:
- vars:
topic: discord issues
along with
function extensionHook(hookName, context) {
if (hookName === "beforeEach") {
counter++;
console.log(`Preparing test ${counter}`);
}
}
module.exports = extensionHook;
and ran:
➜ promptfoo git:(main) ✗ promptfoo eval -c hook/promptfooconfig.yaml --repeat 5 --no-progress-bar
Cache is disabled.
Running 5 concurrent evaluations with 4 threads...
Preparing test 1
Preparing test 2
Preparing test 3
Preparing test 4
Preparing test 5
Do you want beforeEach to run once per var?u1f40e
12/12/2024, 8:56 PMextensions:
- |
if (output != 'beforeEach') return; //:
context.test.vars.pf_tag = Math.floor(Math.random() * Math.pow(36, 6)).toString(36);
The issue is that pf_tag is the same for all 5 repeats of the same test.u1f40e
12/12/2024, 8:57 PMu1f40e
12/12/2024, 9:01 PMmichaelmichaelmichael
12/12/2024, 9:03 PMmichaelmichaelmichael
12/12/2024, 9:12 PMdefaultTest:
options:
transformVars: '{ pf_tag: Math.floor(Math.random() * Math.pow(36, 6)).toString(36)) }'michaelmichaelmichael
12/12/2024, 9:15 PMmichaelmichaelmichael
12/12/2024, 9:16 PMu1f40e
12/12/2024, 10:24 PMmichaelmichaelmichael
12/12/2024, 11:14 PMmichaelmichaelmichael
12/12/2024, 11:14 PMu1f40e
12/12/2024, 11:37 PM--repeat), yeah. That way we can track the variance as well.u1f40e
12/12/2024, 11:38 PMu1f40e
12/12/2024, 11:39 PMmichaelmichaelmichael
12/12/2024, 11:40 PMmichaelmichaelmichael
12/12/2024, 11:41 PM