https://promptfoo.dev/ logo
how to manage prompts in codebase ?
# ❓|questions
c
How would you recommend i manage prompts in my application codebase ? and how do i link them to the evaluations im doing using promptfoo. I think promptfoo is the best tool so far i have come across to help me engineer prompts but im not sure how to go about managing prompts in my codebase , would appreciate any advice.
m
This is a a great question and something that we are actively thinking through right now. Feel free to grab a time on my calendar if you would like to chat about it or have other product feedback: https://cal.com/michael-dangelo/30min
i
As a point of reference, at Discord we typically kept prompts in standalone files (txt templates or json) and imported them in code. This plays nicely with promptfoo's ability to reference standalone files
c
but if i would like break the prompt down to user prompt , system prompt and output template ( the prompt has to output json and i specify the template of the json ). Then the process seems to be quite messy
how i have implemented it so far is i have a prompts_config.json with keys system prompt , user prompt and output template
and then a eval_prompt_gen.py that parses the prompts config and generates the chat completion version of the prompt
m
Just a thought - you can use a custom python or javascript file to contstruct the prompts together - I put this example together yesterday https://github.com/promptfoo/promptfoo/tree/main/examples/claude-vs-gpt-image that may be worth looking at. You could do something similar to define the system, user, and output templates in any way you'd like including in text files.
c
yep thats what i have been doing so far. i implemented a custom python function
m
If you have a specific product idea about how to make this better I am happy to try to implement it for you or iterate on it with you
c
also since my model output is a json. I was wondering how can i set up an eval such that it does a levenshtein on the value of a particular key and the ideal output
m
Where is the ideal output stored?
c
i was thinking maybe it should be in the tests.csv
m
that sounds reasonable to me too
I have been using custom asserts to do things like calculate FAR / FRR + F1 score all in javascript. @ian has been working on making metrics easier to use. He may have better suggestions for you.
I am happy to talk through any of your problems on a call if you'd like to throw time on my calendar. It sounds like you are on a good path. If anything is particularly frusturating please let us know and we will try to fix it for you as quickly as possible
c
yup sure thanks for the quick response! this is a great tool for LLM evaluations will definitely bring up ideas and problems that came up
m
Awesome. I am glad you find it useful. I've used it to solve problems at work that used to take months of engineering effort a team of people doing data annotation in a few days.
2 Views