luiscosio
07/15/2024, 2:57 AMdescription: "My eval"
prompts:
- file://prompts.json
providers:
- id: 'http://localhost:1234/v1/chat/completions'
config:
method: 'POST'
headers:
'Content-Type': 'application/json'
body:
prompt: "{{prompt}}"
responseParser: 'json.choices[0].message.content'
tests:
- vars:
topic: bananas
This is my prompts.json:
[
{
"role": "system",
"content": "You are a helpful assistant"
},
{
"role": "user",
"content": "Tell me about {{topic}}"
}
]
This is the error:
SyntaxError: Bad control character in string literal in JSON at position 82 SyntaxError: Bad control character in string literal in JSON at position 82 at JSON.parse (<anonymous>) at HttpProvider.callApi (C:\Users\Luis\AppData\Roaming\npm\node_modules\promptfoo\dist\src\providers\http.js:37:37) at Evaluator.runEval (C:\Users\Luis\AppData\Roaming\npm\node_modules\promptfoo\dist\src\evaluator.js:297:62) at process.processTicksAndRejections (node:internal/process/task_queues:95:5) at async processEvalStep (C:\Users\Luis\AppData\Roaming\npm\node_modules\promptfoo\dist\src\evaluator.js:621:25) --- SyntaxError: Bad control character in string literal in JSON at position 82 SyntaxError: Bad control character in string literal in JSON at position 82 at JSON.parse (<anonymous>) at HttpProvider.callApi (C:\Users\Luis\AppData\Roaming\npm\node_modules\promptfoo\dist\src\providers\http.js:37:37) at Evaluator.runEval (C:\Users\Luis\AppData\Roaming\npm\node_modules\promptfoo\dist\src\evaluator.js:297:62) at process.processTicksAndRejections (node:internal/process/task_queues:95:5) at async processEvalStep (C:\Users\Luis\AppData\Roaming\npm\node_modules\promptfoo\dist\src\evaluator.js:621:25)
Backend is LM Studio (llama.cpp wrapper) and the model is gemma-2michaelmichaelmichael
07/15/2024, 2:58 AMmichaelmichaelmichael
07/15/2024, 3:05 AM.promptfoo curl http://localhost:1234/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"model": "bartowski/gemma-2-9b-it-GGUF",
"messages": [
{ "role": "system", "content": "Always answer in rhymes." },
{ "role": "user", "content": "Introduce yourself." }
],
"temperature": 0.7,
"max_tokens": -1,
"stream": true
}'
data: {"id":"chatcmpl-vwds46dxt9ga5v39n9vbmr","object":"chat.completion.chunk","created":1721012561,"model":"bartowski/gemma-2-9b-it-GGUF/gemma-2-9b-it-Q4_K_M.gguf","choices":[{"index":0,"delta":{"role":"assistant","content":"I"},"finish_reason":null}]}
data: {"id":"chatcmpl-vwds46dxt9ga5v39n9vbmr","object":"chat.completion.chunk","created":1721012561,"model":"bartowski/gemma-2-9b-it-GGUF/gemma-2-9b-it-Q4_K_M.gguf","choices":[{"index":0,"delta":{"role":"assistant","content":"'"},"finish_reason":null}]}
I can look into adding support for this. Not sure if I will get to it before I go to bed but I will tryluiscosio
07/15/2024, 3:07 AMmichaelmichaelmichael
07/15/2024, 3:09 AMluiscosio
07/15/2024, 3:10 AMluiscosio
07/15/2024, 3:11 AMmichaelmichaelmichael
07/15/2024, 3:16 AMluiscosio
07/15/2024, 3:51 AMmichaelmichaelmichael
07/15/2024, 5:47 AMluiscosio
07/15/2024, 3:59 PMluiscosio
07/15/2024, 3:59 PMmichaelmichaelmichael
07/15/2024, 3:59 PMluiscosio
07/15/2024, 4:00 PMmichaelmichaelmichael
07/15/2024, 4:01 PM