https://promptfoo.dev/ logo
Custom providers python object instantiation
# ❓|questions
p
Hi, quick question around how custom python providers are handled when running test cases. let's say I have an object,
EvalRunner
defined in my custom provider.
EvalRunner
, in its constructor, creates a bunch of instance variables that require a network call.
Copy code
class EvalRunner:
    def __init__(self,):
        self.tools = self._get_tools(key=api_key)
        self.prompts = self._get_prompts(key=api_key)
and in my
call_api
function, I instantiate
EvalRunner
like so:
Copy code
def call_api(prompt, options, context):
    config = options.get("config", None)
    runner = EvalRunner(config=runner_config, model_client=model_client)
im guessing here each time
call_api
runs, I instantiate a new
runner
and make two network calls. if I define runner outside of
call_api
, does that cut down the number of network calls to only one? thank you!
m
If you have a resource that is expensive to set up and don't want to do it multiple times (because for example you download model weights), you could start a process independently that loads your model and then call it separately using http (flask, fastapi), grpc, celery, dask, triton, etc. For example
Copy code
python
from celery import Celery

app = Celery('tasks', broker='redis://localhost:6379/0')

class EvalRunner:
    def __init__(self, api_key):
        self.tools = self._get_tools(key=api_key)
        self.prompts = self._get_prompts(key=api_key)

    def _get_tools(self, key):
        # Simulate a network call
        return f"Tools initialized with key {key}"

    def _get_prompts(self, key):
        # Simulate a network call
        return f"Prompts initialized with key {key}"

    def run(self, prompt):
        return f"Running with prompt: {prompt}"

runner = EvalRunner(api_key="your_api_key")

@app.task
def run_eval(prompt):
    return runner.run(prompt)
and
Copy code
python
from tasks import run_eval

def call_api(prompt, options, context):
    result = run_eval.delay(prompt)
    return result.get(timeout=10)
You would just need to start the celery client
celery -A tasks worker --loglevel=info
(make sure redis is running).
with flask
Copy code
python
from flask import Flask, request, jsonify

app = Flask(__name__)

class EvalRunner:
    def __init__(self, api_key):
        self.tools = self._get_tools(key=api_key)
        self.prompts = self._get_prompts(key=api_key)

    def _get_tools(self, key):
        # Simulate a network call
        return f"Tools initialized with key {key}"

    def _get_prompts(self, key):
        # Simulate a network call
        return f"Prompts initialized with key {key}"

    def run(self, prompt):
        return f"Running with prompt: {prompt}"

# Create an EvalRunner instance with a given API key
runner = EvalRunner(api_key="your_api_key")

@app.route('/run', methods=['POST'])
def run():
    data = request.json
    prompt = data.get('prompt')
    result = runner.run(prompt)
    return jsonify(result=result)

if __name__ == '__main__':
    app.run(port=5000)
Copy code
python
import requests

def call_api(prompt, options, context):
    url = "http://localhost:5000/run"
    payload = {"prompt": prompt}
    response = requests.post(url, json=payload)
    if response.status_code == 200:
        return response.json().get('result')
    else:
        return f"Error: {response.status_code}"
p
is simply setting up the resource outside of
call_api
not sufficient?
Copy code
# Create an EvalRunner instance with a given API key
runner = EvalRunner(api_key="your_api_key")

def call_api(prompt, options, context):
     runner.run(prompt)
m
Unfortunately not, it re-intializes the entire file every time. I'll think about this more and see if we can prototype official support for you.
p
hmmm okay that's good to know. yeah it'd be great if there was official support for this without having to create and run another local app each time I want to run evals
4 Views