What I found after digging deeper:
I explored the codebase and discovered some existing capabilities I initially missed: • The "Edit & Replay" feature in PromptEditor.tsx - allows editing prompts and re-running them • The
/eval-creator
page - a GUI for building configs • Test Case Dialog - add/edit test cases visually
These are great! But they seem oriented around post-eval editing rather than pre-eval exploration.
The specific gap I see:
Current workflow: Write YAML config → Run eval → View results Proposed workflow: Tinker with prompt → See output → Save as test case Basically moving from "Formalize first" to "Explore first".
Why I think this aligns with promptfoo:
Better onboarding - Lower barrier for new users who just want to "try something".
Competitive parity - LangSmith, Humanloop, and PromptLayer all have playground modes.
Doesn't cannibalize Cloud - Enterprise value (team collab, history, compliance) stays intact.
Concrete proposal:
I could extend the existing
/eval-creator
page to add: • A "Run Single Prompt" button (no full eval needed) • Live variable substitution preview • "Mark as Test Case" with auto-suggested assertions • Export to
promptfooconfig.yaml
Before I start:
A few questions to avoid wasted effort:
Is this direction something you'd accept as a PR, or would you prefer it as a standalone community tool?
Any architectural preferences? (e.g., should it reuse the existing
runAssertion
pipeline or be separate?)
Who's the right person to loop in for design review?