https://promptfoo.dev/ logo
Playground Effort?
# ❓|questions
t
Hey team! 👋 I've been exploring promptfoo for LLM evaluation for the past few hours, but I'm hitting some friction in my workflow and wanted to get your thoughts. The friction: - The current workflow requires manually editing YAML to create/tweak tests - There's no "playground" mode to quickly iterate on prompts before formalizing them as tests - I find myself wanting to tinker with a prompt, see the output, then decide if it should become a test case - Basically: explore → validate → formalize (vs the current: formalize → validate) What I'm imagining: A UI layer where I can: - Edit prompts with live variable substitution preview - Run single prompts against a model on-the-fly - Mark good outputs as "save as test case" with auto-generated assertions - Export everything to promptfooconfig.yaml when I'm done tinkering My questions: 1. Does something like this already exist in promptfoo that I'm missing? (The web UI seems read-only for results) 2. Is this kind of interactive "prompt tinkering" layer consistent with promptfoo's design philosophy, or is the code-first approach intentional? 3. If I were to build this, would you recommend: - A standalone tool that exports to promptfoo format - A fork/PR to add this to the existing web UI Happy to contribute if this aligns with where the project is heading. Just don't want to duplicate effort or go against the grain. Happy New Year 💌
What I found after digging deeper: I explored the codebase and discovered some existing capabilities I initially missed: • The "Edit & Replay" feature in PromptEditor.tsx - allows editing prompts and re-running them • The /eval-creator page - a GUI for building configs • Test Case Dialog - add/edit test cases visually These are great! But they seem oriented around post-eval editing rather than pre-eval exploration. The specific gap I see: Current workflow: Write YAML config → Run eval → View results Proposed workflow: Tinker with prompt → See output → Save as test case Basically moving from "Formalize first" to "Explore first". Why I think this aligns with promptfoo: Better onboarding - Lower barrier for new users who just want to "try something". Competitive parity - LangSmith, Humanloop, and PromptLayer all have playground modes. Doesn't cannibalize Cloud - Enterprise value (team collab, history, compliance) stays intact. Concrete proposal: I could extend the existing /eval-creator page to add: • A "Run Single Prompt" button (no full eval needed) • Live variable substitution preview • "Mark as Test Case" with auto-suggested assertions • Export to promptfooconfig.yaml Before I start: A few questions to avoid wasted effort: Is this direction something you'd accept as a PR, or would you prefer it as a standalone community tool? Any architectural preferences? (e.g., should it reuse the existing runAssertion pipeline or be separate?) Who's the right person to loop in for design review?
2 Views