https://promptfoo.dev/ logo
Output is empty
# ❓|questions
h
I'm following the guide here: https://www.promptfoo.dev/docs/configuration/testing-llm-chains#using-a-custom-provider and when I run the
callApi
function manually via Node, it works. However, when I run it with promptfoo, it just returns an empty output. Any idea why this might be happening?
i
would you be able to share the custom provider implementation?
and the relevant part of the promptfoo config
h
Here's the config:
Copy code
yaml
prompts:
  - "Please send the user different messages depending on when the app was last used. Before you start, please ask me any questions you have about this so I can give you more context."
providers: ["./assistant.js"]
and here's the custom provider:
Copy code
js
/* eslint-disable @typescript-eslint/no-var-requires */
const { spawn } = require("child_process");
const { join } = require("path");

// eslint-disable-next-line no-undef -- __dirname is a Node.js global
const dashboardRoot = join(__dirname, "..");
const pathToRspec = join(dashboardRoot, "{path-to-my-file}");
const RE = /(?<=PROMPTFOO START)(.*?)(?=PROMPTFOO END)/gms;

class ChainProvider {
  id() {
    return "liquid-assistant";
  }

  async callApi(prompt) {
    return new Promise((resolve, reject) => {
      const rbProcess = spawn("bundle", ["exec", "rspec", pathToRspec], {
        cwd: dashboardRoot,
        // eslint-disable-next-line no-undef -- process.env is a Node.js global
        env: { ...process.env, PROMPTFOO: true, PROMPTFOO_PROMPT: prompt },
      });

      let output;

      rbProcess.stdout.on("data", (data) => {
        if (!data) {
          return;
        }

        output += data.toString();
      });

      rbProcess.on("close", (code) => {
        if (code !== 0) {
          reject(`ruby script exited with code ${code}`);
        } else {
          output = output.match(RE).join().trim();

          resolve({ output });
        }
      });
    });
  }
}

module.exports = ChainProvider;
i
I don't see anything obviously wrong here, wondering if you've tried adding debug outputs and running the eval with the
LOG_LEVEL=debug
envar set? Alternatively, you could try using a custom script provider, which essentially does what you are trying to do by shelling out to ruby:
h
Let me try that and lyk
I ran it with
LOG_LEVEL=debug
and got this:
Copy code
At least one evaluation failure occurred. This might be caused by the underlying call to the provider, or a test failure. Context:
[{"provider":{"id":"liquid-assistant"},"prompt":{"raw":"Please send the user different messages depending on when the app was last used. Before you start, please ask me any questions you have about this so I can give you more context.","label":"Please send the user different messages depending on when the app was last used. Before you start, please ask me any questions you have about this so I can give you more context."},"vars":{},"response":{"output":""},"success":false,"score":0,"namedScores":{},"latencyMs":12977,"error":"No output"}]
which doesn't feel very helpful. Is it possible it's timing out?
I have noticed that it takes longer when it runs all the way via node than it does on promptfoo
i
there's no timeout (or caching) built into custom providers. I think it's more likely it's getting a nullish string somehow
maybe we can add some logs to the custom provider to see what's coming through?
h
Yeah it's definitely doing something weird with the API call. Here it is when I run the JS file directly in Node:
Copy code
sh
Knapsack adapter is off!

Run options: include {:focus=>true}


All examples were filtered out; ignoring {:focus=>true}



LiquidAssistantController

  #liquid_assistant


PROMPTFOO START
<some valid string>
PROMPTFOO END

    works with promptfoo

Finished in 12.22 seconds (files took 6.57 seconds to load)
1 example, 0 failures
and here it is inside promptfoo:
Copy code
sh
Knapsack adapter is off!

Run options: include {:focus=>true}


All examples were filtered out; ignoring {:focus=>true}


LiquidAssistantController

  #liquid_assistant

Run options: include {:focus=>true}


All examples were filtered out; ignoring {:focus=>true}


LiquidAssistantController

  #liquid_assistant


PROMPTFOO START

PROMPTFOO END

    works with promptfoo


Finished in 5.01 seconds (files took 8.48 seconds to load)
1 example, 0 failures
note that the correct one took 12 seconds whereas the one in promptfoo took 5s. It's almost as if the API call isn't happening 😕
i
🤔
h
Is it doing anything with env vars?
i
maybe check what envars are being sent?
h
It appears that the promptfoo run gets all the env vars that running it directly in Node does. It does add some extras though, namely
npm_*
style env vars that aren't there by default.
I can't imagine that would cause this issue though.
Ok another development! When I use this prompt:
Say hi to the user including their name.
it totally works.
When I use my original prompt above, same error.
I think I figured it out! It had to do with the way I have things set up on my end. Thanks for your help!
i
Ah that's good to know. I was stumped. Is there anything that promptfoo could have done differently to make it easier to debug?
h
I think maybe don't automatically fail on empty output? For context, I was passing (the wrong) one of two LLM generated texts, which should have been an empty string given the prompt.
On a separate note, I'm getting this error now:
Copy code
llm-rubric produced malformed response: {
  "pass": true,
  "reason": "The 'message' field includes a greeting with 'Hi there!' or 'Hi {{first_name_var}}!' depending on the condition."
}
now sure what's malformed here
i
hmm
looking at the code, I think there is an issue in how it handles reasons that contain braces {}
I'll put in a fix for that. As a temp workaround, is there a way you can remove the {{ }} ?
h
Not really, I'm specifically testing whether the generated output has the correct kind of templating syntax
i
alrighty. yeah the code tries to handle some failure cases such as when the llm includes extra text before/after the actual json. But in this case, it winds up slicing on the wrong closing brace
this is a small fix, probably will have it in in a day
h
Appreciate it 🙏
i
another potential workaround. switch from
llm-rubric
to
model-graded-closedqa
. the behavior is similar, but it won't have this issue
PR for llm-rubric:
h
That worked!
Last question (hopefully): is there a way to run the tests in sequence rather than in parallel?
i
Yep pass
-j 1
as an option to the eval cli to reduce concurrency to 1
h
Ok sorry final final question (but maybe not lol): is there a way to only run certain tests for specific prompts?
i
hmm, you can filter tests by description using
--filer-pattern <pattern>
but not on a per-prompt basis. For that unfortunately you'd have to break out into different test suites
5 Views