https://promptfoo.dev/ logo
Concurrent execution of python providers
# ❓|questions
v
Our prompts are pretty complex, so to get around it, we load our input vars dynamically in the provider, as dumping all of the data to the test cases would be massive. Here is an example of our current execution setup: promptfooconfig.yaml
Copy code
yaml
prompts: prompts.txt
providers: exec:python provider.py
tests: tests.yaml
tests.yaml
Copy code
yaml
- vars:
    id: 1
  assert:
    - type: contains
      value: a
    - type: contains-json
- vars:
    id: 2
  assert:
    - type: contains
      value: b
    - type: contains-json

...
provider.py
Copy code
py
...
kwargs = json.loads(sys.argv[3])["vars"]
id = kwargs["id"]
with app.app_context():
    # gathers vars from db, makes api calls for other data, etc
    prompt_vars = gather_prompt_vars(id=id)
    chain = Chain(vars=prompt_vars, use_cache=False)
    chain(print_response=True)
Our issue is that running our tests.yaml contains 1000+ test cases. We've tried running with
-j 5
to set the max concurrency, but that seems to be specifically for API calls and not running providers concurrently. Is there any way to run the provider in parallel here? Currently our execution is push 120+ min...
i
hmm, multiple test cases should run in parallel even with a single provider. Are you only seeing a single invocation of
provider.py
at any given time?
The only exception to this would be if you are using the
delay
parameter or the
_conversation
variable (which it doesn't sound like you're doing)