https://promptfoo.dev/ logo
Batch inference support?
# ❓|questions
d
Hey folks, loving my experience with promptfoo so far! My one main issue so far is inference speed; I've built a custom Python provider that makes inference requests to an on-premise instance of Mixtral-8x22b. In my experiments, I'm running inference on 100s of samples with a few different prompts, and I'm finding that these inference requests are running 5-10x slower than I would've expected if promptfoo were batching inference requests. I'm curious -- does promptfoo run batch inference under the hood, and if not, is there any way to configure this? Ideally I'd like to run inference requests with a minimum batch_size of ~20. Would really appreciate any support folks are able to provide!
i
👀 promptfoo doesn’t batch inference under the hood (ie. doesn’t put multiple requests in the same api call) but it does parallelize individual requests. Maybe you can increase the concurrency?
d
Thanks for your quick response Ian! Should be fine that multiple requests aren't placed in the same api call -- how do I increase the concurrency?
i
the
-j
or
--max-concurrency
param will do that. default is 4 at a time, but if your instance can handle more then go for it!
Copy code
promptfoo eval -j 4
d
Amazing, will try that out -- thanks Ian!
Is this the correct way to include max_concurrency in my config?
Copy code
description: romantic_prompts_100concurrency
prompts:
- file://romantic_prompts_1_24/*.txt
providers:
- id: python:inference.py
  config:
    llm_type: mistralai/Mixtral-8x22B-Instruct-v0.1
tests:
- tests/romantic_prompts_100concurrency/dataset.csv
max_concurrency: 100
Not noticing any latency improvement unfortunately
i
in the config, it would be
Copy code
yaml
evaluateOptions:
  maxConcurrency: 100
()
2 Views