https://promptfoo.dev/ logo
custom provider - Can't make single minimal custom...
# ❓|questions
a
I followed the instruction, but can't create fully minimal example that is working.
Copy code
description: 'My first eval'


prompts:
  - "Write a tweet about {{topic}}"
  - "Write a very concise, funny tweet about {{topic}}"

# providers: [ollama:mistral]

providers: 
  - test_script.py:
      id: 'python3:test_script.py'
      config:
        temperature: 0
          # config:
  #   model: claude-v2
  #   temperature: 0
  #   max_tokens: 50

tests:
  - vars:
      topic: bananas

  - vars:
      topic: avocado toast
    assert:
      # For more information on assertions, see https://promptfoo.dev/docs/configuration/expected-outputs
      - type: icontains
        value: avocado
      - type: javascript
        value: 1 / (output.length + 1)
and
Copy code
import json

def call_api(prompt, options, context):
    # The 'options' parameter contains additional configuration for the API call.
    additional_option = options.get('additionalOption', None)

    # The 'context' parameter provides info about which vars were used to create the final prompt.
    user_variable = context['vars'].get('userVariable', None)

    # The prompt is the final prompt string after the variables have been processed.
    # Custom logic to process the prompt goes here.
    # For instance, you might call an external API or run some computations.
    output = "yes"


    # The result should be a dictionary with at least an 'output' field.
    result = {
        "output": output,
    }

    # if some_error_condition:
    #     result['error'] = "An error occurred during processing"

    # if token_usage_calculated:
    #     # If you want to report token usage, you can set the 'tokenUsage' field.
    #     result['tokenUsage'] = {"total": token_count, "prompt": prompt_token_count, "completion": completion_token_count}

    return result
got "SyntaxError: Cannot use import statement outside a module"
Any instruction to create working example step-by-step would be great!
(The current docs, are not that clear)
i
try setting it up like this:
Copy code
yaml
providers:
 - id: 'python:test_script.py'
   config:
      temperature: 0
you're getting that error in part because
python3
is not a valid provider name. It's just
python
. You can specify the
PROMPTFOO_PYTHON
environment variable if you want to point it to a specific python binary