https://promptfoo.dev/ logo
Testing a prompt that produces multiple determinis...
# ❓|questions
e
Hi! I am creating a minimal LLM prompt to help me sort my shopping lists. I use an app with existing recipes to produce the raw shopping list, then use a prompt to take out of the list the common items (like salt, pepper, etc.). My goal is to be able to share the raw list from my phone share button and get in the output the list sorted out. This is the prompt I am using at the moment:
Copy code
You are a helpful assistant that processes shopping lists. Your task is to take an unstructured list of ingredients and categorize them, removing common items that are likely already present in a typical kitchen. Follow these steps:
Parse the given list of ingredients.
Categorize the ingredients based on the provided headings (in all caps).
Remove common kitchen items such as salt, pepper, water, garlic, onions, basic spices, etc.
Create a new list in markdown format with the following structure:
Main heading: "Shopping list"
Use the categories from the original list as second-level headings
List ingredients that need to be purchased under their respective categories
Create a "Check these" category for items that might already be in the kitchen but should be verified (e.g., lemons, garlic)
Add a "Removed ingredients" section at the bottom for items that were removed from the list
Ensure that the output follows this exact format:
```markdown
# Shopping list
## [Category 1]
Ingredient 1
Ingredient 2
## [Category 2]
Ingredient 3
Ingredient 4
...

# Check these
Item 1
Item 2

# Removed ingredients
Common item 1
Common item 2

Here's the list of ingredients to process:

[previous list]

Please process this list and provide the output in the specified markdown format.
``` How do you deal with these kinds of evals in promptfoo? My initial plan is to create a separate prompt for each of the top level headings and then apply deterministic evals for contains not-contains. Is there any better way to do it? (I read about JSON but it's not clear how to verify if subtrees match)
My goal is to evaluate how different models perform this particular task. I would start just with three raw shopping lists and fine tune the test for each example before trying to generalize the evaluations
i
I think your plan sounds good. I always try to lean toward deterministic when possible
6 Views