https://promptfoo.dev/ logo
Using CSV for Vars
# ❓|questions
r
Hello, I am trying to wrap my head around how to accomplish the following. > Use a CSV with rows of data to be used as vars. The csv is my labeled data that I want to essentially assert the LLM with. Meaning that I expect the LLM to answer close to exactly as the labeled data has in the csv. promptfooconfig.yaml
Copy code
prompts: prompts.txt

providers:
  - exec:python file://gemini-prompt.py

tests:
  - vars: sample_5.csv
  - assert:
      - type: equals
        value: '{{expected_client_name}}'
        transform: output.client_name
gemini-prompt.py
Copy code
chain = prompt | model | schema.output_parser

document = sys.argv[1]
print("document:", document)


print(chain.invoke({"text": document}).json())
Within the
gemini-prompt.py
I am expecting it to receive values from each row of my
sampe_5.csv
so that my function can use them . Instead I am seeing the vars get broken up into key value pairs....
Copy code
Error: Command failed: python file  {"config":{"basePath":""}} {"vars":{"0":"d","1":"o","2":"c","3":"_","4":"r","5":"e","6":"f","7":"e","8":"r
Any help is appreciated.
Bump 🙏
I've made a little progress but still not quite there. gemini-prompt.py - This is not returning anything though, I keep getting the error
No output
in promptfoo. But if I print
result
there is indeed a response from the LLM , just not being passed to promptfoo.
Copy code
import asyncio
import json
import sys

import pandas as pd
from dotenv import load_dotenv
from langchain.prompts import PromptTemplate
from langchain_google_vertexai import VertexAI

import schema as schema
from global_utils.prompt_manager import delete_prompt, fetch_prompt, store_prompt

load_dotenv()
model = VertexAI(model_name="gemini-pro")


def fetch_map_prompt():
    template, input_vars, partial_vars = fetch_prompt(
        "DocAI Provisional Debt Extraction"
    )
    return PromptTemplate(
        template=template, input_variables=input_vars, partial_variables=partial_vars
    )


async def process_prompt(prompt_sys):
    prompt = fetch_map_prompt()

    chain = prompt | model | schema.provisional_output_parser

    placeholder_doc = """
    Client Name Adam Hilgendorf
    """

    result = await chain.ainvoke({"text": placeholder_doc})
    # print("result_llm:", result.json())
    return result


if __name__ == "__main__":
    result = asyncio.run(process_prompt(sys.argv[1]))
    result.json()
promptfooconfig.yaml - Ideally my asserts would be done outside of the csv file but I just couldn't figure out how to do that.
Copy code
prompts: prompts.txt

providers:
  # - vertex:gemini-pro
  - exec:python gemini-prompt.py

tests: promptfoo_evals_pd(5).csv
s
It looks like your
gemini-prompt.py
provider script is missing the required
call_api
function. I believe that function name specifically is what's called when promptfoo is evaluating. Given that you're missing that function, I think your results aren't getting passed through to promptfoo. Try creating a function with that name with the required params that outputs the required output schema documented here: https://www.promptfoo.dev/docs/providers/python
r
Thank you! For reference the code below is working now. gemini-prompt.py
Copy code
from dotenv import load_dotenv
from langchain.prompts import PromptTemplate
from langchain_google_vertexai import VertexAI

import schema as schema
from global_utils.prompt_manager import delete_prompt, fetch_prompt, store_prompt

load_dotenv()
model = VertexAI(model_name="gemini-pro")


def call_api(prompt, options, context):
    # Fetch map prompt
    template, input_vars, partial_vars = fetch_prompt(
        "DocAI Provisional Debt Extraction"
    )
    prompt = PromptTemplate(
        template=template, input_variables=input_vars, partial_variables=partial_vars
    )

    # Process prompt
    chain = prompt | model | schema.provisional_output_parser
    placeholder_doc = """
    Client Name Adam Hilgendorf
    """
    result = chain.invoke({"text": placeholder_doc})
    result_dict = result.__dict__

    print("result_llm", result_dict)
    # Return result
    return {"output": result_dict}
promptfooconfig.yaml
Copy code
prompts: prompts.txt

providers:
  # - vertex:gemini-pro
  - id: python:gemini-prompt.py

tests: promptfoo_evals_pd(5).csv
For the promptfooconfig.yaml, is there anyway I can use asserts along side the imported vars from my csv? Right now I have them in the csv
__expected
but I'd like to avoid that if possible.
s
Just to be clear, you want to include the expected value and have your python provider script take that in and use it alongside the LLM output during the evaluation, correct?
r
Here is the assertion I have in the csv right now for reference
Copy code
fn: output.client_name == "Adam Hilgendorf";
s
I assume this means there are several expected values, one for each input. Is that correct?
r
I would like to take the response from the python provider and run an assertion on it using a var from the csv
I tried this but it did not work...
Copy code
- assert:
      - type: equals
        value: '{{expected_client_name}}'
        transform: output.client_name
expected_client_name
is column in my csv and for that row contains a name.
output.client_name
is the result from the llm
At least that's how I was thinking about it
s
Are you able to audio chat? I'd love to ask some more clarifying questions.
r
sure thing
s
FR sent. Think we need to be friends to chat given that there are only text channels here.
Why do you want to avoid using the __expected column?
Is it just that you don’t like that column name or is there another reason?
r
there are hundreds of these evaluations and I'd rather not have to type the out each time is the main reason lol
s
Can you give another example of an eval you’d want to run other than the equals expected field?
r
Most of my evals would be to check for equality between the llm response and the data in the csv. I have several fields I need to do this on so I would have several
__expected
columns, I believe. Most are equality checks but one or two might be semantic similarity or something custom like a python function to compare last 4 digits.
New question here.... I have a
call_api
function that is working to interact with the llm. Now I am trying to run my assertions that are in my
__expected
columns within my
tests.csv
The function
call_api
returns json in string representation and I return that in my ProviderResponse:
Copy code
prompt = PromptTemplate(
        template=template, input_variables=input_vars, partial_variables=partial_vars
    )

    # Process prompt
    chain = prompt | model | schema.provisional_output_parser

    result = chain.invoke({"text": page_contents}).json()
    # print("raw_llm_result", result)

    # Return result
    return {"output": result}
Result looks like a flat object:
Copy code
{"client_name": "John Doe", etc...}
An assertion in the
__expected
column in my csv looks like:
Copy code
fn:output.client_name == context.vars.client_name;
When I look in the web viewer I can see the
output
is showing my JSON but what I can't see is the values my assertion used, it just says :
Copy code
Custom function returned false
 output.client_name == context.vars.client_name;
I'm not certain that either of the two values I'm checking for equality are actually being populated correctly.
i
I would set up the vars like:
Copy code
yaml
tests:
  - vars:
      filepath: sample_5.csv
    assert:
      - type: equals
        # ...
(note that you have an error in your original config,
assert
should go under the vars key and not have its own
-
) and the
gemini-prompt.py
file has to have a function named
call_api
. The file path will be passed in
context['vars']['filepath']
See https://promptfoo.dev/docs/providers/python#python-script-1 for an example and other reference information about the
call_api
arguments
a
Hello! Did you ever figure out how to "wrap" a column with an assert, as you were looking to do in the quoted message? We have a spreadsheet with several thousand rows used by multiple teams, so are trying to achieve the same thing: asserts based on column data, without having to transform existing columns into
__expected
. Would love to know if you solved it! Cheers
3 Views