vitmont
03/28/2024, 5:12 PMyaml
prompts: prompts.txt
providers: exec:python provider.py
tests: tests.yaml
tests.yaml
yaml
- vars:
id: 1
assert:
- type: contains
value: a
- type: contains-json
- vars:
id: 2
assert:
- type: contains
value: b
- type: contains-json
...
provider.py
py
...
kwargs = json.loads(sys.argv[3])["vars"]
id = kwargs["id"]
with app.app_context():
# gathers vars from db, makes api calls for other data, etc
prompt_vars = gather_prompt_vars(id=id)
chain = Chain(vars=prompt_vars, use_cache=False)
chain(print_response=True)
Our issue is that running our tests.yaml contains 1000+ test cases. We've tried running with -j 5 to set the max concurrency, but that seems to be specifically for API calls and not running providers concurrently. Is there any way to run the provider in parallel here? Currently our execution is push 120+ min...ian
03/28/2024, 5:41 PMprovider.py at any given time?ian
03/28/2024, 5:47 PMdelay parameter or the _conversation variable (which it doesn't sound like you're doing)