https://promptfoo.dev/ logo
Implementing the similar assertion
# ❓|questions
o
Here is my input "Magic is an ancient art that captivates audiences by defying logic and creating wonder. From sleight of hand tricks to grand illusions, magicians use skill, creativity, and showmanship to entertain. Magic often blends storytelling with mystery, drawing people into a world where the impossible becomes possible. READ MORE: The History of Magic and Its Greatest Illusions Throughout history, it has evolved, but its essence remains a celebration of imagination and surprise." my prompt is trying to remove this line of text "READ MORE: The History of Magic and Its Greatest Illusions " How do I check the output of the LLM and validate I have the right result? In my test.csv file I have the following columns "original_text", "output_text", "_expected1". The column _expected1 has the following: similar(0.9) + output_text However I don't think these asserts are being applied correctly. Even when I insert this mock assert similar(0.9):Hello world which should fail but this does not. What am I missing?
m
Can you share more of your config including a row or two of your csv? You could use a
contains
assert or a
not-contains
assert on READ MORE. You could also set up a model graded like
llm-rubric
Copy code
assert:
  - type: llm-rubric
    value: Does not contain READ MORE
I am also happy to jump on a quick call if you want to work on it toegther https://cal.com/michael-dangelo/15min?date=2024-09-16&month=2024-09
o
Here is my config file: prompts: - id: file://prompts/prompt_V1.txt label: prompt_1 providers: - id: openai:gpt-4o-mini config: temperature: 0 apiKey: apikey tests: - file://clean_text_test_cases.csv The prompt is basically asking the model to clean / get rid of any unrelated text. Data row example, In addition to dressing for the weather, it is also important to wear clothing that is appropriate for the activity. Magnet fishing and using magnet fishing kits with the throw and pull technique can be a physically demanding activity, and it is important to wear clothing that allows you to move freely and comfortably. Avoid wearing loose or baggy clothing that could get caught on debris or obstruct your movement. It is also a good idea to wear sturdy, closed-toe shoes to protect your feet from sharp objects or slippery surfaces. READ Best Places To Magnet Fish Near Me In this case I am trying to remove "READ Best Places To Magnet Fish Near Me" This can be dynamic meaning these text can be many types or varied. The output from the LLM will be only "In addition to dressing for the weather, it is also important to wear clothing that is appropriate for the activity. Magnet fishing and using magnet fishing kits with the throw and pull technique can be a physically demanding activity, and it is important to wear clothing that allows you to move freely and comfortably. Avoid wearing loose or baggy clothing that could get caught on debris or obstruct your movement. It is also a good idea to wear sturdy, closed-toe shoes to protect your feet from sharp objects or slippery surfaces." or similar. Hence Iam trying to use the similar assert to check.
m
I figured it out! Your csv had _expected1 instead of __expected1 (two underscores). I am going to put a warning in promptfoo to catch errors like this in the future