https://promptfoo.dev/ logo
beforeEach w/ --repeat
# ❓|questions
u
Is there a way to run code before each test with
--repeat
. Currently
beforeEach
runs once per input, rather than 5 times per input with
--repeat 5
. Alternately, is there any unique per-test identifier we can access from the UI? I'm trying to push a unique id through our provider so we can search for traces specific to each test.
m
Hi @u1f40e I am not sure if I understand your issue. I set up
Copy code
description: "Before Each Repeat Test"
prompts:
  - "Write a tweet about: {{topic}}."
extensions:
  - file://hooks.js:extensionHook
providers:
  - openai:chat:gpt-4o-mini
tests:
  - vars:
      topic: discord issues
along with
Copy code
function extensionHook(hookName, context) {
    if (hookName === "beforeEach") {
        counter++;
        console.log(`Preparing test ${counter}`);
    }
}

module.exports = extensionHook;
and ran:
Copy code
➜  promptfoo git:(main) ✗ promptfoo eval -c hook/promptfooconfig.yaml --repeat 5 --no-progress-bar
Cache is disabled.
Running 5 concurrent evaluations with 4 threads...
Preparing test 1
Preparing test 2
Preparing test 3
Preparing test 4
Preparing test 5
Do you want beforeEach to run once per var?
u
I'd like to be able to generate a different tag for each of the repeats of a test. This is my extension in this case:
Copy code
extensions:
  - |
    if (output != 'beforeEach') return; //:
    context.test.vars.pf_tag = Math.floor(Math.random() * Math.pow(36, 6)).toString(36);
The issue is that
pf_tag
is the same for all 5 repeats of the same test.
(that code will output a schema error, but it works anyways)
Maybe the vars are shared between all the repeats?
m
thanks for context. looking
Out of curiosity, where did you get the idea to use a hook for this? a hook relies on a side effect using something like transformVars which was designed for mutating vars
Copy code
defaultTest:
  options:
    transformVars: '{ pf_tag: Math.floor(Math.random() * Math.pow(36, 6)).toString(36)) }'
repeat is designed to exactly repeat the same input which is why your vars are the same. The repeat logic happens around here: https://github.com/promptfoo/promptfoo/blob/main/src/evaluator.ts#L577 I am trying to think of an elegant way to accomplish your goal without breaking the behavior for other users. What are you using pf_tag for?
feel free to grab some time on my calendar if you would prefer to talk in a call https://cal.com/michael-dangelo/15min
u
Oh thanks. I can put it in transformVars if that works. I started using hooks because I can have multiple of them. I think defaultTest can only have one transformVars. I'm passing pf_tag to an HTTP API and then logging it to our tracing (in Phoenix). So now we can search phoenix for the tag to quickly debug quality issues.
m
You can also consider unrolling your test suite. If your tests are defined in an external file (*.yaml) you can specify that 5 times or just prepend an external id outside of promptfoo
happy to brainstorm strategies with you
u
For some use cases, we are going to run N times (without
--repeat
), yeah. That way we can track the variance as well.
Our tests are spread across many files, composed by including multiple configs on the CLI, and assembled via hooks and transforms
I can show you what we're doing sometime if you're interested
m
Do you have a few minutes now?