Is there a way to take multiple LLM responses and send them to another LLM which can then do some action using all of the responses? For example, if I run a prompt through three LLMs so I now have 3 responses. Can I send those 3 responses to another LLM and ask it to combine the three or some other action using all three. I see there is a "select best" metric but I don't want to choose one, I want to combine 3 responses in some fashion. Thank you for your help.

Yes, you can do this using

storeOutputAs

to chain outputs between tests. Use

storeOutputAs

to save each model's response, then reference them in a final test that combines all three:

yaml
prompts:
  - "{{query}}"

providers:
  - openai:gpt-5-nano

tests:
  - vars: {query: "What is quantum computing?"}
    provider: openai:gpt-5-nano
    options:
      storeOutputAs: response1

  - vars: {query: "What is quantum computing?"}
    provider: anthropic:messages:claude-4.5-haiku
    options:
      storeOutputAs: response2

  - vars: {query: "What is quantum computing?"}
    provider: google:gemini-2.5-flash
    options:
      storeOutputAs: response3

  # Combine all three
  - vars:
      query: |
        Combine these three responses:
        Model 1: {{response1}}
        Model 2: {{response2}}
        Model 3: {{response3}}

When you use

storeOutputAs

, promptfoo automatically runs tests sequentially so variables are available to later tests. Note: Tests run one at a time (not parallel) when using this feature.

@michaelmichaelmichael AWESOME! Thank you so much. I will give this a try. Is there any way to reference this in a python wrapper? Like if I want to take the result of the combination and feed it into some other system?

prompts:
  - "{{query}}"

defines a prompt template. in this case it is a just a pass through to the vars, but many people will have a prefix or system prompt there. You can rename it to whatever you like or even omit it entirely.

vars: {query: "What is quantum computing?"}

could also be:

vars: {prompt: "What is quantum computing?"}

with

`prompts:
  - "{{prompt}}"

for example. it's a nunjucks string. As for passing from a python wrapper. Yes you can use https://www.promptfoo.dev/docs/configuration/expected-outputs/python/ within a config or you can even

import subprocess
  import json

  # Run promptfoo
  result = subprocess.run(
      ['npx', 'promptfoo', 'eval', '-c', 'config.yaml', '-o', '/dev/stdout'],
      capture_output=True,
      text=True
  )

  data = json.loads(result.stdout)

depending on what you are trying to do. LLMs + coding agents can be very useful thinking through options here.

I am using my company's internal LLMs that work with an openai api interface. It seem to only be able to work if I include the info directly in the providers section like this:

providers:
  - id: openai:chat:openai/gpt-oss-120b
    config:
      apiHost: "models.k8s.aip.company.com"
      apiKey: "sk-abcfoo"

This works fine. Once I try what you mention above about combining outputs where I put the provider in vars then I start getting errors. > - vars: {query: "What is quantum computing?"} > provider: openai:chat:openai/gpt-oss-120b Any idea how I can get this to work? I've been trying for a day or so and still have been getting errors. Usually the error is: > Failed to sanitize URL models.k8s.aip.company.com/chat/completions: TypeError: Invalid URL

@storied Hey, looks like your issue is unrelated to your original issue- the apiHost you've set doesn't include the protocol (i.e. the

https://

part of the url), hence the "Invalid URL" issue. You'll need to set that correctly to proceed.

@[Promptfoo] Faizan Thanks for the response. When I add https:// it seems to double the protocol of the url. That's why I removed it. See below. > Request to https://https://models.k8s.aip.company.com/v1/chat/completions failed (attempt #3), retrying: TypeError: fetch failed (Cause: Error: getaddrinfo ENOTFOUND https)

@storied do you have a typo? I notice it seems to be missing a

:

in the second

https//

?

No typo. For some reason when I paste this message in Discord I lose one :. I don't know why. But the error message says https://https:// with a colon here

Ah understood- this only happens with multiple responses

@[Promptfoo] Faizan Here is an example. Thanks so much for your help with this. https://cdn.discordapp.com/attachments/1429928857842221239/1440350217080799422/judge-cl.yaml?ex=692710ac&is=6925bf2c&hm=c296503639ef67f08f28430cc39dd3380f320c43f5cbd8e08711a5e4481e264a&