abhagsain
04/01/2024, 8:28 PM.env file and added this
env
PROMPTFOO_REMOTE_API_BASE_URL=https://foo.example.com
PROMPTFOO_REMOTE_APP_BASE_URL=https://foo.example.com
Now I'm running npx promptfoo@latest eval. It creates an eval, but when I run pf view -y, it opens it in my local browser and not in the server, and when I see the server, there are no evals.
How do I send evals from my local to the self-hosted version?ian
04/01/2024, 8:54 PMpromptfoo share, it should send it over (assuming you run from same directory with the .env file)ian
04/01/2024, 8:54 PMabhagsain
04/01/2024, 8:57 PMabhagsain
04/01/2024, 8:58 PM_npx/81bbc6515d992ace/node_modules/node-fetch/lib/index.js:1501
reject(new FetchError(`request to ${request.url} failed, reason: ${err.message}`, 'system', err));
^
FetchError: request to https://foo.example.com/api/eval failed, reason: connect ETIMEDOUT 2606:4700:3033::ac43:c659:443
at ClientRequest.<anonymous> (/Users/mac/.npm/_npx/81bbc6515d992ace/node_modules/node-fetch/lib/index.js:1501:11)
at ClientRequest.emit (node:events:527:28)
at TLSSocket.socketErrorListener (node:_http_client:454:9)
at TLSSocket.emit (node:events:527:28)
at emitErrorNT (node:internal/streams/destroy:151:8)
at emitErrorCloseNT (node:internal/streams/destroy:116:3)
at process.processTicksAndRejections (node:internal/process/task_queues:82:21) {
type: 'system',
errno: 'ETIMEDOUT',
code: 'ETIMEDOUT'
}ian
04/01/2024, 8:59 PMabhagsain
04/01/2024, 9:04 PMian
04/01/2024, 9:07 PMabhagsain
04/01/2024, 9:38 PMts
pf eval --verbose --no-cache --table
🚀 - renderPrompt - prompt: { default: [Function (anonymous)] }
🚀 - renderPrompt - prompt: {
raw: '[object Object]',
display: '[object Object]',
function: { default: [Function (anonymous)] }
}
🚀 - renderPrompt - prompt: { default: [Function (anonymous)] }
/Users/mac/.nvm/versions/node/v18.0.0/lib/node_modules/promptfoo/dist/src/evaluator.js:141
const result = await prompt.function({ vars });
^
TypeError: prompt.function is not a function
at renderPrompt (/Users/mac/.nvm/versions/node/v18.0.0/lib/node_modules/promptfoo/dist/src/evaluator.js:141:45)
at Evaluator.runEval (/Users/mac/.nvm/versions/node/v18.0.0/lib/node_modules/promptfoo/dist/src/evaluator.js:214:38)
at /Users/mac/.nvm/versions/node/v18.0.0/lib/node_modules/promptfoo/dist/src/evaluator.js:569:36
This is my prompt.ts file
ts
import dedent from "dedent";
export default async ({
vars,
}: {
vars: any;
}): Promise<
{
role: "system" | "user" | "assistant";
content: string;
}[]
> => {
return [
{
role: "system",
content: dedent`You're a helpful assistant`,
},
{
role: "user",
content: dedent`Hello`,
},
{
role: "assistant",
content: dedent`Hello`,
},
{
role: "user",
content: dedent`${vars.question}`,
},
];
};abhagsain
04/01/2024, 9:38 PMabhagsain
04/01/2024, 9:41 PMyaml
description: "Test eval"
prompts: [dist/prompt.js]
providers:
- id: azureopenai:chat:gpt3-16k
config:
apiHost: server-openai.openai.azure.com
temperature: 0.4
apiVersion: 2024-02-15-preview
tests:
- vars:
question: What's the weather like today?
assert:
- type: javascript
value: file://test-file.jsabhagsain
04/01/2024, 9:56 PM0.49.3 and it worked (but I don't have the newest feature) so the issue is most likely with the latest version.ian
04/01/2024, 10:00 PM.default due to the es6 export. There was a related change in this area.
I should be able to publish a fix in the next few hoursian
04/01/2024, 11:49 PMdist/prompt.js so I can test the fix?abhagsain
04/02/2024, 7:49 AMjs
"use strict";
var __awaiter = (this && this.__awaiter) || function (thisArg, _arguments, P, generator) {
function adopt(value) { return value instanceof P ? value : new P(function (resolve) { resolve(value); }); }
return new (P || (P = Promise))(function (resolve, reject) {
function fulfilled(value) { try { step(generator.next(value)); } catch (e) { reject(e); } }
function rejected(value) { try { step(generator["throw"](value)); } catch (e) { reject(e); } }
function step(result) { result.done ? resolve(result.value) : adopt(result.value).then(fulfilled, rejected); }
step((generator = generator.apply(thisArg, _arguments || [])).next());
});
};
var __importDefault = (this && this.__importDefault) || function (mod) {
return (mod && mod.__esModule) ? mod : { "default": mod };
};
Object.defineProperty(exports, "__esModule", { value: true });
const dedent_1 = __importDefault(require("dedent"));
exports.default = (_a) => __awaiter(void 0, [_a], void 0, function* ({ vars: { noOfSlides, text, additionalInfo = "" }, }) {
return [
{
role: "system",
content: (0, dedent_1.default) `You're a helpful assistant`,
},
];
});ian
04/02/2024, 12:43 PMabhagsain
04/02/2024, 2:44 PMian
04/02/2024, 4:36 PMabhagsain
04/02/2024, 5:04 PMnpx promptfoo@latest and It's not picking up the path to the prompt file now. I'm always getting
pf eval --verbose --no-cache
Cache is disabled.
You must provide at least 1 prompt
This is the YAML file. I switched to 0.50.0 it works but with the previous error and I changed the value in the node_modules and it's working fine.
yaml
description: "Summarize texts "
prompts: [dist/summarize-prompt.js]abhagsain
04/02/2024, 5:15 PMian
04/02/2024, 6:00 PMdist/prompt.js you provided 😱😱ian
04/02/2024, 6:02 PMlabel. For example:
yaml
providers:
- id: azureopenai:chat:gpt3-16k
label: 'gpt3-16k high temperature'
config:
apiHost: server-openai.openai.azure.com
temperature: 0.4
apiVersion: 2024-02-15-previewabhagsain
04/02/2024, 6:11 PMian
04/02/2024, 6:13 PMyaml
prompts:
- file://dist/prompt.js
providers:
- openai:gpt-3.5-turbo-0613
tests:
- vars:
topic: fooian
04/02/2024, 6:13 PMian
04/02/2024, 6:13 PMpromptfoo eval in the same directory as the configabhagsain
04/02/2024, 6:17 PM