https://promptfoo.dev/ logo
Does Ollama stream responses when running evals?
# ❓|questions
e
I am using Ollama to evaluate several models using promptfoo. When running the evals using the verbose flag, I get to see several individual tokens being sent from the Ollama server. Checking the docs from Ollama, you can suppress streaming setting the stream parameter to false. I have used this configuration but the output is streamed (making no difference). Disabling stream was added around one year ago in Ollama: https://github.com/ollama/ollama/pull/639
Copy code
yaml
providers:
  - id: ollama:chat:llama3.1:8b-instruct-q8_0
    label: llama31_8B_Q8
    config:
      temperature: 0
      stream: false
      system: "You are an expert in foobar test generation ..."
Sample console output when verbose is enabled:
Copy code
Ollama generate API response: {"model":"llama3.1:8b-instruct-q8_0","created_at":"2024-09-02T12:49:16.348475547Z","message":{"role":"assistant","content":"No"},"done":false}
{"model":"llama3.1:8b-instruct-q8_0","created_at":"2024-09-02T12:49:16.36468283Z","message":{"role":"assistant","content":" tengo"},"done":false}
{"model":"llama3.1:8b-instruct-q8_0","created_at":"2024-09-02T12:49:16.38080782Z","message":{"role":"assistant","content":" información"},"done":false}
{"model":"llama3.1:8b-instruct-q8_0","created_at":"2024-09-02T12:49:16.396949072Z","message":{"role":"assistant","content":" sobre"},"done":false}
{"model":"llama3.1:8b-instruct-q8_0","created_at":"2024-09-02T12:49:16.413265688Z","message":{"role":"assistant","content":" las"},"done":false}
Note that this behavior is also observed in for the grader LLM.
Extra question: Would you rather have this filled as a Github issue if confirmed?
m
Hey @elsatch, a github issue is great! I will look into disabling streaming mode by default for you today.
e
Great, I'll post the info there. One thing to consider: It might be that the Ollama endpoint is not getting the config parameter at all
i
@elsatch I disabled streaming. it appears the
config
parameter in promptfoo is treated as the ollama
options
parameter ☠️ That's why
stream
didn't work
e
I imagined it was something along these lines. Thanks for the update!