- I think we can use Git to store stable prompt(not updated frequently), and langfuse(langsmith) can be used to need to manage prompt in runtime with versions.. also DB/registry can be used for dynamic prompts and sometimes prompts can be embedded in code
- the prompts can be tested directly are isolated prompt with exact input/output, for example "summarize this article, extract contact info from this email", etc.. And the one can be only evaluated in context are multi-task prompts need to use tools, memory, user intent analysis, etc
- I think this seems to be really difficult part, so in this case, some metrics like success rate, retries, tool errors can be used. human feedback is also a good eval metric. and teams can use their own scoring system for evals
- Yeah, I can say using these combinations becaue I use langfuse for registry and Git for version control, RAG for dynamic block search, E2E evals by human feedback and dry-run agents(simulation).. also planning analysis pipeline based on metrics like success/error rate, retries when the platform is on production