richm88
02/27/2024, 2:08 AMprompts: prompts.txt
providers:
- exec:python file://gemini-prompt.py
tests:
- vars: sample_5.csv
- assert:
- type: equals
value: '{{expected_client_name}}'
transform: output.client_name
gemini-prompt.py
chain = prompt | model | schema.output_parser
document = sys.argv[1]
print("document:", document)
print(chain.invoke({"text": document}).json())
Within the gemini-prompt.py I am expecting it to receive values from each row of my sampe_5.csv so that my function can use them . Instead I am seeing the vars get broken up into key value pairs....
Error: Command failed: python file {"config":{"basePath":""}} {"vars":{"0":"d","1":"o","2":"c","3":"_","4":"r","5":"e","6":"f","7":"e","8":"r
Any help is appreciated.richm88
02/27/2024, 11:19 PMrichm88
02/28/2024, 12:01 AMNo output in promptfoo. But if I print result there is indeed a response from the LLM , just not being passed to promptfoo.
import asyncio
import json
import sys
import pandas as pd
from dotenv import load_dotenv
from langchain.prompts import PromptTemplate
from langchain_google_vertexai import VertexAI
import schema as schema
from global_utils.prompt_manager import delete_prompt, fetch_prompt, store_prompt
load_dotenv()
model = VertexAI(model_name="gemini-pro")
def fetch_map_prompt():
template, input_vars, partial_vars = fetch_prompt(
"DocAI Provisional Debt Extraction"
)
return PromptTemplate(
template=template, input_variables=input_vars, partial_variables=partial_vars
)
async def process_prompt(prompt_sys):
prompt = fetch_map_prompt()
chain = prompt | model | schema.provisional_output_parser
placeholder_doc = """
Client Name Adam Hilgendorf
"""
result = await chain.ainvoke({"text": placeholder_doc})
# print("result_llm:", result.json())
return result
if __name__ == "__main__":
result = asyncio.run(process_prompt(sys.argv[1]))
result.json()
promptfooconfig.yaml - Ideally my asserts would be done outside of the csv file but I just couldn't figure out how to do that.
prompts: prompts.txt
providers:
# - vertex:gemini-pro
- exec:python gemini-prompt.py
tests: promptfoo_evals_pd(5).csvsirtylerdurden.
02/28/2024, 12:26 AMgemini-prompt.py provider script is missing the required call_api function. I believe that function name specifically is what's called when promptfoo is evaluating. Given that you're missing that function, I think your results aren't getting passed through to promptfoo. Try creating a function with that name with the required params that outputs the required output schema documented here: https://www.promptfoo.dev/docs/providers/pythonrichm88
02/28/2024, 1:56 AMfrom dotenv import load_dotenv
from langchain.prompts import PromptTemplate
from langchain_google_vertexai import VertexAI
import schema as schema
from global_utils.prompt_manager import delete_prompt, fetch_prompt, store_prompt
load_dotenv()
model = VertexAI(model_name="gemini-pro")
def call_api(prompt, options, context):
# Fetch map prompt
template, input_vars, partial_vars = fetch_prompt(
"DocAI Provisional Debt Extraction"
)
prompt = PromptTemplate(
template=template, input_variables=input_vars, partial_variables=partial_vars
)
# Process prompt
chain = prompt | model | schema.provisional_output_parser
placeholder_doc = """
Client Name Adam Hilgendorf
"""
result = chain.invoke({"text": placeholder_doc})
result_dict = result.__dict__
print("result_llm", result_dict)
# Return result
return {"output": result_dict}
promptfooconfig.yaml
prompts: prompts.txt
providers:
# - vertex:gemini-pro
- id: python:gemini-prompt.py
tests: promptfoo_evals_pd(5).csvrichm88
02/28/2024, 1:57 AM__expected but I'd like to avoid that if possible.sirtylerdurden.
02/28/2024, 2:01 AMrichm88
02/28/2024, 2:01 AMfn: output.client_name == "Adam Hilgendorf";sirtylerdurden.
02/28/2024, 2:01 AMrichm88
02/28/2024, 2:03 AMrichm88
02/28/2024, 2:05 AM- assert:
- type: equals
value: '{{expected_client_name}}'
transform: output.client_name
expected_client_name is column in my csv and for that row contains a name. output.client_name is the result from the llmrichm88
02/28/2024, 2:05 AMsirtylerdurden.
02/28/2024, 2:05 AMrichm88
02/28/2024, 2:05 AMsirtylerdurden.
02/28/2024, 2:07 AMsirtylerdurden.
02/28/2024, 2:14 AMsirtylerdurden.
02/28/2024, 2:15 AMrichm88
02/28/2024, 2:16 AMsirtylerdurden.
02/28/2024, 2:19 AMrichm88
02/28/2024, 2:21 AM__expected columns, I believe. Most are equality checks but one or two might be semantic similarity or something custom like a python function to compare last 4 digits.richm88
02/29/2024, 12:39 AMcall_api function that is working to interact with the llm. Now I am trying to run my assertions that are in my __expected columns within my tests.csv
The function call_api returns json in string representation and I return that in my ProviderResponse:
prompt = PromptTemplate(
template=template, input_variables=input_vars, partial_variables=partial_vars
)
# Process prompt
chain = prompt | model | schema.provisional_output_parser
result = chain.invoke({"text": page_contents}).json()
# print("raw_llm_result", result)
# Return result
return {"output": result}
Result looks like a flat object:
{"client_name": "John Doe", etc...}
An assertion in the __expected column in my csv looks like:
fn:output.client_name == context.vars.client_name;
When I look in the web viewer I can see the output is showing my JSON but what I can't see is the values my assertion used, it just says :
Custom function returned false
output.client_name == context.vars.client_name;
I'm not certain that either of the two values I'm checking for equality are actually being populated correctly.ian
03/01/2024, 6:46 PMyaml
tests:
- vars:
filepath: sample_5.csv
assert:
- type: equals
# ...
(note that you have an error in your original config, assert should go under the vars key and not have its own -)
and the gemini-prompt.py file has to have a function named call_api. The file path will be passed in context['vars']['filepath']
See https://promptfoo.dev/docs/providers/python#python-script-1 for an example and other reference information about the call_api argumentsalexclaydon
05/01/2024, 7:51 AM__expected. Would love to know if you solved it! Cheers