halsdunes
06/25/2024, 7:57 PMcallApi function manually via Node, it works. However, when I run it with promptfoo, it just returns an empty output. Any idea why this might be happening?ian
06/25/2024, 7:59 PMian
06/25/2024, 7:59 PMhalsdunes
06/25/2024, 8:02 PMyaml
prompts:
- "Please send the user different messages depending on when the app was last used. Before you start, please ask me any questions you have about this so I can give you more context."
providers: ["./assistant.js"]
and here's the custom provider:
js
/* eslint-disable @typescript-eslint/no-var-requires */
const { spawn } = require("child_process");
const { join } = require("path");
// eslint-disable-next-line no-undef -- __dirname is a Node.js global
const dashboardRoot = join(__dirname, "..");
const pathToRspec = join(dashboardRoot, "{path-to-my-file}");
const RE = /(?<=PROMPTFOO START)(.*?)(?=PROMPTFOO END)/gms;
class ChainProvider {
id() {
return "liquid-assistant";
}
async callApi(prompt) {
return new Promise((resolve, reject) => {
const rbProcess = spawn("bundle", ["exec", "rspec", pathToRspec], {
cwd: dashboardRoot,
// eslint-disable-next-line no-undef -- process.env is a Node.js global
env: { ...process.env, PROMPTFOO: true, PROMPTFOO_PROMPT: prompt },
});
let output;
rbProcess.stdout.on("data", (data) => {
if (!data) {
return;
}
output += data.toString();
});
rbProcess.on("close", (code) => {
if (code !== 0) {
reject(`ruby script exited with code ${code}`);
} else {
output = output.match(RE).join().trim();
resolve({ output });
}
});
});
}
}
module.exports = ChainProvider;ian
06/25/2024, 8:05 PMhalsdunes
06/25/2024, 8:06 PMhalsdunes
06/25/2024, 8:08 PMLOG_LEVEL=debug and got this:
At least one evaluation failure occurred. This might be caused by the underlying call to the provider, or a test failure. Context:
[{"provider":{"id":"liquid-assistant"},"prompt":{"raw":"Please send the user different messages depending on when the app was last used. Before you start, please ask me any questions you have about this so I can give you more context.","label":"Please send the user different messages depending on when the app was last used. Before you start, please ask me any questions you have about this so I can give you more context."},"vars":{},"response":{"output":""},"success":false,"score":0,"namedScores":{},"latencyMs":12977,"error":"No output"}]
which doesn't feel very helpful. Is it possible it's timing out?halsdunes
06/25/2024, 8:10 PMian
06/25/2024, 8:10 PMian
06/25/2024, 8:11 PMhalsdunes
06/25/2024, 8:16 PMsh
Knapsack adapter is off!
Run options: include {:focus=>true}
All examples were filtered out; ignoring {:focus=>true}
LiquidAssistantController
#liquid_assistant
PROMPTFOO START
<some valid string>
PROMPTFOO END
works with promptfoo
Finished in 12.22 seconds (files took 6.57 seconds to load)
1 example, 0 failures
and here it is inside promptfoo:
sh
Knapsack adapter is off!
Run options: include {:focus=>true}
All examples were filtered out; ignoring {:focus=>true}
LiquidAssistantController
#liquid_assistant
Run options: include {:focus=>true}
All examples were filtered out; ignoring {:focus=>true}
LiquidAssistantController
#liquid_assistant
PROMPTFOO START
PROMPTFOO END
works with promptfoo
Finished in 5.01 seconds (files took 8.48 seconds to load)
1 example, 0 failures
note that the correct one took 12 seconds whereas the one in promptfoo took 5s. It's almost as if the API call isn't happening 😕ian
06/25/2024, 8:17 PMhalsdunes
06/25/2024, 8:53 PMian
06/25/2024, 8:54 PMhalsdunes
06/25/2024, 9:19 PMnpm_* style env vars that aren't there by default.halsdunes
06/25/2024, 9:19 PMhalsdunes
06/25/2024, 9:33 PMSay hi to the user including their name. it totally works.halsdunes
06/25/2024, 9:33 PMhalsdunes
06/26/2024, 3:08 PMian
06/26/2024, 3:47 PMhalsdunes
06/26/2024, 4:26 PMhalsdunes
06/26/2024, 4:26 PMllm-rubric produced malformed response: {
"pass": true,
"reason": "The 'message' field includes a greeting with 'Hi there!' or 'Hi {{first_name_var}}!' depending on the condition."
}
now sure what's malformed hereian
06/26/2024, 4:30 PMian
06/26/2024, 4:30 PMian
06/26/2024, 4:31 PMhalsdunes
06/26/2024, 4:33 PMian
06/26/2024, 4:34 PMian
06/26/2024, 4:35 PMhalsdunes
06/26/2024, 4:36 PMian
06/26/2024, 4:38 PMllm-rubric to model-graded-closedqa. the behavior is similar, but it won't have this issuehalsdunes
06/26/2024, 5:06 PMhalsdunes
06/26/2024, 5:17 PMian
06/26/2024, 5:21 PM-j 1 as an option to the eval cli to reduce concurrency to 1halsdunes
06/26/2024, 5:49 PMian
06/27/2024, 3:51 AM--filer-pattern <pattern> but not on a per-prompt basis. For that unfortunately you'd have to break out into different test suites