adminini
02/24/2024, 10:58 PMdescription: 'My first eval'
prompts:
- "Write a tweet about {{topic}}"
- "Write a very concise, funny tweet about {{topic}}"
# providers: [ollama:mistral]
providers:
- test_script.py:
id: 'python3:test_script.py'
config:
temperature: 0
# config:
# model: claude-v2
# temperature: 0
# max_tokens: 50
tests:
- vars:
topic: bananas
- vars:
topic: avocado toast
assert:
# For more information on assertions, see https://promptfoo.dev/docs/configuration/expected-outputs
- type: icontains
value: avocado
- type: javascript
value: 1 / (output.length + 1)
and import json
def call_api(prompt, options, context):
# The 'options' parameter contains additional configuration for the API call.
additional_option = options.get('additionalOption', None)
# The 'context' parameter provides info about which vars were used to create the final prompt.
user_variable = context['vars'].get('userVariable', None)
# The prompt is the final prompt string after the variables have been processed.
# Custom logic to process the prompt goes here.
# For instance, you might call an external API or run some computations.
output = "yes"
# The result should be a dictionary with at least an 'output' field.
result = {
"output": output,
}
# if some_error_condition:
# result['error'] = "An error occurred during processing"
# if token_usage_calculated:
# # If you want to report token usage, you can set the 'tokenUsage' field.
# result['tokenUsage'] = {"total": token_count, "prompt": prompt_token_count, "completion": completion_token_count}
return result got "SyntaxError: Cannot use import statement outside a module"adminini
02/24/2024, 11:00 PMadminini
02/24/2024, 11:00 PMian
02/25/2024, 1:05 AMyaml
providers:
- id: 'python:test_script.py'
config:
temperature: 0
you're getting that error in part because python3 is not a valid provider name. It's just python. You can specify the PROMPTFOO_PYTHON environment variable if you want to point it to a specific python binaryian
02/25/2024, 1:06 AM