hephaestus_klytotekhnes
01/09/2025, 10:10 PMpromptfoo -j $n) is used, it looks like the API calls are being batched by-test rather than by-provider. This means, with Ollama, since it simply handles calls in-order, that it's juggling models in an out of memory for literally every single call.
In this specific situation, the fast way to run an eval would be to run all of the calls to a given provider, and then all of the calls for the next provider, and so on. Is there a way to do this?
( My workaround in the short term has been to simply comment out all-but-one provider, then run the same eval multiple subsequent times with only a single provider each time. Then, I can do a final run with all the providers uncommented, which will leverage the cache and finish almost immediately, to produce a combined eval report. )