https://promptfoo.dev/ logo
Issues with LM Studio and gemma-2
# ❓|questions
l
This is my config file:
Copy code
description: "My eval"

prompts: 
  - file://prompts.json

providers:
  - id: 'http://localhost:1234/v1/chat/completions'
    config:
      method: 'POST'
      headers:
        'Content-Type': 'application/json'
      body: 
        prompt: "{{prompt}}"
      responseParser: 'json.choices[0].message.content'
  

tests:
  - vars:
      topic: bananas
This is my prompts.json:
Copy code
[
    {
        "role": "system",
        "content": "You are a helpful assistant"
    },
    {
        "role": "user",
        "content": "Tell me about {{topic}}"
    }
]
This is the error:
Copy code
SyntaxError: Bad control character in string literal in JSON at position 82 SyntaxError: Bad control character in string literal in JSON at position 82 at JSON.parse (<anonymous>) at HttpProvider.callApi (C:\Users\Luis\AppData\Roaming\npm\node_modules\promptfoo\dist\src\providers\http.js:37:37) at Evaluator.runEval (C:\Users\Luis\AppData\Roaming\npm\node_modules\promptfoo\dist\src\evaluator.js:297:62) at process.processTicksAndRejections (node:internal/process/task_queues:95:5) at async processEvalStep (C:\Users\Luis\AppData\Roaming\npm\node_modules\promptfoo\dist\src\evaluator.js:621:25) --- SyntaxError: Bad control character in string literal in JSON at position 82 SyntaxError: Bad control character in string literal in JSON at position 82 at JSON.parse (<anonymous>) at HttpProvider.callApi (C:\Users\Luis\AppData\Roaming\npm\node_modules\promptfoo\dist\src\providers\http.js:37:37) at Evaluator.runEval (C:\Users\Luis\AppData\Roaming\npm\node_modules\promptfoo\dist\src\evaluator.js:297:62) at process.processTicksAndRejections (node:internal/process/task_queues:95:5) at async processEvalStep (C:\Users\Luis\AppData\Roaming\npm\node_modules\promptfoo\dist\src\evaluator.js:621:25)
Backend is LM Studio (llama.cpp wrapper) and the model is gemma-2
m
Thanks for starting a question. I'm downloading the model now through LM Studio. I will let you know what I find out.
ok I think this is related to how LM studio tries to stream a response which we do not currently support
Copy code
.promptfoo curl http://localhost:1234/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "bartowski/gemma-2-9b-it-GGUF",
    "messages": [
      { "role": "system", "content": "Always answer in rhymes." },
      { "role": "user", "content": "Introduce yourself." }
    ],
    "temperature": 0.7,
    "max_tokens": -1,
    "stream": true
}'
data: {"id":"chatcmpl-vwds46dxt9ga5v39n9vbmr","object":"chat.completion.chunk","created":1721012561,"model":"bartowski/gemma-2-9b-it-GGUF/gemma-2-9b-it-Q4_K_M.gguf","choices":[{"index":0,"delta":{"role":"assistant","content":"I"},"finish_reason":null}]}

data: {"id":"chatcmpl-vwds46dxt9ga5v39n9vbmr","object":"chat.completion.chunk","created":1721012561,"model":"bartowski/gemma-2-9b-it-GGUF/gemma-2-9b-it-Q4_K_M.gguf","choices":[{"index":0,"delta":{"role":"assistant","content":"'"},"finish_reason":null}]}
I can look into adding support for this. Not sure if I will get to it before I go to bed but I will try
l
I would appreciate it a lot, I'll be around ironing out other parts of my pipeline, please let me know if there is anything else you need from me
m
What's your github username? I'll tag you in the PR
l
luiscosio
m
it is not streaming related but I did figure it out. I'll open a PR shortly
l
Thank you so much!
m
https://github.com/promptfoo/promptfoo/pull/1156 fixes it. We'll merge and deploy it in the morning
l
@michaelmichaelmichael
Works just fine
m
Awesome!
l
I'll start working on the arc prompts to force gemma to produce JSON, and if anything let you know, and thank you again
m
Sure! Please let me know how it goes. Happy to build more features for you.
3 Views