https://promptfoo.dev/ logo
How are you guys managing prompts in practice?
# ❓|questions
m
How are you guys managing prompts in practice? I’m realizing prompt management is not one problem. Some prompts are easy to store/version in tools like PromptLayer or similar registries, and some can be evaluated with straightforward tests. But a lot of prompts seem harder to evaluate because their quality only shows up in a real workflow, with full context, tools, and downstream effects. In those cases, the judgment can be pretty subjective and closer to an end-to-end or integration test than a unit test. So I’m curious how people are actually handling this in real systems: • Where do you store/version different kinds of prompts? • How do you decide which prompts can be tested directly vs only evaluated in-context? • For subjective prompts like code review, planning, or UI generation, what does “good enough” evaluation look like for you? • Are you using prompt registries, Git, RAG, human review, E2E evals, or some combination?
a
in the past, I mainly used langfuse and langsmith for prompt management and it was great
m
@Azlan why'd you stop?
a
@mrlanderson actually, there is no eval pipeline in langfuse or langsmith but I'd like to share my thoughts for prompt evals
- I think we can use Git to store stable prompt(not updated frequently), and langfuse(langsmith) can be used to need to manage prompt in runtime with versions.. also DB/registry can be used for dynamic prompts and sometimes prompts can be embedded in code - the prompts can be tested directly are isolated prompt with exact input/output, for example "summarize this article, extract contact info from this email", etc.. And the one can be only evaluated in context are multi-task prompts need to use tools, memory, user intent analysis, etc - I think this seems to be really difficult part, so in this case, some metrics like success rate, retries, tool errors can be used. human feedback is also a good eval metric. and teams can use their own scoring system for evals - Yeah, I can say using these combinations becaue I use langfuse for registry and Git for version control, RAG for dynamic block search, E2E evals by human feedback and dry-run agents(simulation).. also planning analysis pipeline based on metrics like success/error rate, retries when the platform is on production
2 Views