https://promptfoo.dev/ logo
Is there a way to batch by-provider, rather than b...
# ❓|questions
h
I'm using a single Ollama host which can only have one model in memory at a time, but I'm using multiple ollama providers to specify multiple models for comparison. Regardless of what specific concurrency factor (
promptfoo -j $n
) is used, it looks like the API calls are being batched by-test rather than by-provider. This means, with Ollama, since it simply handles calls in-order, that it's juggling models in an out of memory for literally every single call. In this specific situation, the fast way to run an eval would be to run all of the calls to a given provider, and then all of the calls for the next provider, and so on. Is there a way to do this? ( My workaround in the short term has been to simply comment out all-but-one provider, then run the same eval multiple subsequent times with only a single provider each time. Then, I can do a final run with all the providers uncommented, which will leverage the cache and finish almost immediately, to produce a combined eval report. )