https://promptfoo.dev/ logo
Have you seen LLM outputs change shape after promp...
# ❓|questions
a
Examples: - JSON object -> quoted JSON string - scalar enum -> explanatory sentence - tool call / next step changes unexpectedly Do you usually catch this with promptfoo/schema assertions/logs, or is it still painful to debug? I’m sanity-checking whether this is a real problem outside my own workflows.
i
Yeah these are not unusual if you're changing between models and prompts etc. I generally catch these and prevent regressions using promptfoo. You can run these tests in development or even in CI/CD
2 Views