Embfan
06/19/2024, 7:12 AMian
06/19/2024, 7:14 AMEmbfan
06/19/2024, 8:30 AMmanojlds
06/20/2024, 11:28 AMmanojlds
06/21/2024, 8:37 AMAlex Gagliano
06/25/2024, 5:22 PMprompts:
- '{{prompt}}'
providers:
- id: 'localhost:8000/ollama/api/chat'
config:
method: 'POST'
headers:
'Authorization': 'Bearer ...'
'Content-Type': 'application/json'
'Accept': 'application/json'
body:
model: 'speakyse-5k'
stream: 'false'
messages: [{"role": "user", "content": "{{prompt}}"}]
responseParser: 'json'
defaultTest:
options:
providers:
- id: 'localhost:8000/ollama/api/chat'
config:
method: 'POST'
headers:
'Authorization': 'Bearer ...'
'Content-Type': 'application/json'
'Accept': 'application/json'
body:
model: 'speakyse-5k'
stream: 'false'
messages: [{"role": 'user', "content":"{{prompt}}"}]
responseParser: 'json'
tests:
- vars:
prompt: Please walk me through {{topic}}, being as specific as possible.
topic: pulsational pair-instability supernovae
assert:
- type: llm-rubric
value: reponse must be brief, detailed, and precise
When I drop the tests, I no longer get this error (eval table populates with the output without issue). Do you know what's going wrong? Thanks!
EDIT: Looks like this might be specific to llm-rubric?raxrb
07/05/2024, 10:25 AMraxrb
07/05/2024, 10:25 AMraxrb
07/05/2024, 10:34 AMraxrb
07/05/2024, 10:34 AMraxrb
07/05/2024, 10:34 AMian
07/06/2024, 4:44 AMyaml
tests:
- vars:
topic: foobar
assert:
- type: levenshtein
threshold: 2
value: '{{foobar}}'
yubo
07/07/2024, 1:56 AMluiscosio
07/15/2024, 2:11 AM# Learn more about building a configuration: https://promptfoo.dev/docs/configuration/guide
description: "My eval"
prompts:
- '{{prompt}}'
providers:
- id: 'http://localhost:1234/v1/chat/completions'
config:
method: 'POST'
headers:
'Content-Type': 'application/json'
body:
messages: [{"role": "user", "content": "{{prompt}}"}]
responseParser: 'json.choices[0].message.content'
tests:
- vars:
topic: bananas
- vars:
topic: avocado toast
assert:
# For more information on assertions, see https://promptfoo.dev/docs/configuration/expected-outputs
# Make sure output contains the word "avocado"
- type: icontains
value: avocado
# Prefer shorter outputs
- type: javascript
value: 1 / (output.length + 1)
- vars:
topic: new york city
assert:
# For more information on model-graded evals, see https://promptfoo.dev/docs/configuration/expected-outputs/model-graded
- type: llm-rubric
value: ensure that the output is funny
And I'm my API i'm receivng this error:
[2024-07-14 20:07:19.708] [INFO] Received POST request to /v1/chat/completions with body: {
"messages": [
{
"role": "user",
"content": ""
}
]
}
[2024-07-14 20:07:19.708] [ERROR] [Server Error] {"title":"'messages' array must only contain objects with a 'content' field that is not empty"}
luiscosio
07/15/2024, 2:12 AMluiscosio
07/15/2024, 2:18 AMluiscosio
07/15/2024, 2:28 AMluiscosio
07/15/2024, 2:32 AM# Learn more about building a configuration: https://promptfoo.dev/docs/configuration/guide
description: "My eval"
prompts:
- file://prompts.json
providers:
- id: 'http://localhost:1234/v1/chat/completions'
config:
method: 'POST'
headers:
'Content-Type': 'application/json'
body:
prompt: "{{prompt}}"
responseParser: 'json.choices[0].message.content'
tests:
- vars:
topic: bananas
- vars:
topic: avocado toast
assert:
# For more information on assertions, see https://promptfoo.dev/docs/configuration/expected-outputs
# Make sure output contains the word "avocado"
- type: icontains
value: avocado
# Prefer shorter outputs
- type: javascript
value: 1 / (output.length + 1)
- vars:
topic: new york city
assert:
# For more information on model-graded evals, see https://promptfoo.dev/docs/configuration/expected-outputs/model-graded
- type: llm-rubric
value: ensure that the output is funny
luiscosio
07/15/2024, 2:32 AM[
{
"role": "system",
"content": "You are a helpful assistant"
},
{
"role": "user",
"content": "Tell me about {{topic}}"
}
]
luiscosio
07/15/2024, 2:33 AMmichaelmichaelmichael
07/15/2024, 2:48 AMluiscosio
07/15/2024, 2:49 AMmichaelmichaelmichael
07/15/2024, 2:49 AMluiscosio
07/15/2024, 2:50 AMluiscosio
07/15/2024, 2:50 AMluiscosio
07/15/2024, 2:52 AMmichaelmichaelmichael
07/15/2024, 2:53 AMluiscosio
07/15/2024, 2:53 AMluiscosio
07/15/2024, 2:57 AMneoteristis
07/19/2024, 8:48 AM