Hi folks, I have deployed a decent (system) prompts of multiple agents in an agentic chat app to my test environment. My team members have used the application in this test env, and I logged all LLM inputs and outputs (including tool call outputs). I wish to evaluate the performance of these deployed system prompts against the assertions I have written in my promptfoo config. Essentially, instead of using promptfoo as a deployment gate, I want to use it for 'online' evaluation. promptfoo evaluates the "output" of the prompt against the assertions, can I "override" this output without making any LLM calls?

I think this would probably require some custom code wrapping the promptfoo node package (https://www.promptfoo.dev/docs/usage/node-package/). I would probably write something that can just evaluate a test case on demand, and you can feed your inputs/outputs to the test case by an http endpoint or however you wanna access it in your online system. Does that make sense?

Thanks @[Promptfoo] Faizan , I shall take a look at the package! At the moment here is my workaround, which may help someone else:

yaml
prompts:
  - just say "hello"
defaultTest:
  assert:
    - test 1
    - test 2
  options:
    provider:
      id: azure:chat:gpt-4.1
    transform: file://judgement.py:replace_output
tests:
  - path: file://judgement.py:create_tests
providers:
  - id: azure:chat:gpt-4o-mini # Use the cheapest provider, coz all we need is a hello.

I generate test cases programmatically (its just a list of [role, content] messages). I include the "output" from my logs as another var (called

output

) Sample test case:

json
{
  "vars": {
    "messages": [
      {
        "role": "user",
        "content": "some utterance"
      },
      {
        "role": "assistant",
        "content": {}
    ],
    "output": {
      "response": "something that was observed from the prod LLM output",
      "other field": "whatever",
    },
  },
  "description": "acid=51, turn=1, total=2, intent_agent",
  "metadata": {
    "conversationId": "01K7469T9BWCPRV690ERPRNBGM",
    "turn": 1,
    "turns": 2,
    "relevance": 0.0,
    "score": 0.0,
    "agent_name": "intent_agent",
  }
}

I use

messages

and other vars in my LLM-as-judge assertions. Here is the one-liner transform function that replaces the LLM output with the var

output

py
# judgement.py
def replace_output(output, context) -> str:
    return context['vars']['output']