https://promptfoo.dev/ logo
When using self hosted, the npx promptfoo doesn't ...
# ❓|questions
a
I have set up a self-hosted version of PF, thinking I'll be able to send all the evals from my local or CI to the hosted version and share it with other team members. I have added a
.env
file and added this
Copy code
env
PROMPTFOO_REMOTE_API_BASE_URL=https://foo.example.com
PROMPTFOO_REMOTE_APP_BASE_URL=https://foo.example.com
Now I'm running
npx promptfoo@latest eval
. It creates an eval, but when I run
pf view -y
, it opens it in my local browser and not in the server, and when I see the server, there are no evals. How do I send evals from my local to the self-hosted version?
i
If you run
promptfoo share
, it should send it over (assuming you run from same directory with the .env file)
Can you give that a try?
a
it's getting stuck. I received some axios timeout error. Let me see reproduce it and share the exact error.
Copy code
_npx/81bbc6515d992ace/node_modules/node-fetch/lib/index.js:1501
            reject(new FetchError(`request to ${request.url} failed, reason: ${err.message}`, 'system', err));
                   ^
FetchError: request to https://foo.example.com/api/eval failed, reason: connect ETIMEDOUT 2606:4700:3033::ac43:c659:443
    at ClientRequest.<anonymous> (/Users/mac/.npm/_npx/81bbc6515d992ace/node_modules/node-fetch/lib/index.js:1501:11)
    at ClientRequest.emit (node:events:527:28)
    at TLSSocket.socketErrorListener (node:_http_client:454:9)
    at TLSSocket.emit (node:events:527:28)
    at emitErrorNT (node:internal/streams/destroy:151:8)
    at emitErrorCloseNT (node:internal/streams/destroy:116:3)
    at process.processTicksAndRejections (node:internal/process/task_queues:82:21) {
  type: 'system',
  errno: 'ETIMEDOUT',
  code: 'ETIMEDOUT'
}
i
Looks like it is respecting your environment variables and trying to request the remote API. Are you sure the service is running on that side?
a
Yes the service is running but I have put it behind Cloudflare Zero Trust for auth. I can view it in the browser after logging in but I'm checking if that's causing the issue for API calls.
i
ah yeah seems likely
a
It's taking too much time to debug, so I'll keep it on hold for now. But there's another issue in the latest version (it was working fine yesterday). We're using TS for our app and I wanted to keep using TS for writing the prompts as well. So I'm writing prompt in TS and passing the dist/prompt.js to the promptconfig I have added a console log in the node_module file, here's the output
Copy code
ts
pf eval --verbose --no-cache --table

🚀 - renderPrompt - prompt: { default: [Function (anonymous)] }
🚀 - renderPrompt - prompt: {
  raw: '[object Object]',
  display: '[object Object]',
  function: { default: [Function (anonymous)] }
}
🚀 - renderPrompt - prompt: { default: [Function (anonymous)] }
/Users/mac/.nvm/versions/node/v18.0.0/lib/node_modules/promptfoo/dist/src/evaluator.js:141
        const result = await prompt.function({ vars });
                                            ^

TypeError: prompt.function is not a function
    at renderPrompt (/Users/mac/.nvm/versions/node/v18.0.0/lib/node_modules/promptfoo/dist/src/evaluator.js:141:45)
    at Evaluator.runEval (/Users/mac/.nvm/versions/node/v18.0.0/lib/node_modules/promptfoo/dist/src/evaluator.js:214:38)
    at /Users/mac/.nvm/versions/node/v18.0.0/lib/node_modules/promptfoo/dist/src/evaluator.js:569:36
This is my prompt.ts file
Copy code
ts
import dedent from "dedent";

export default async ({
  vars,
}: {
  vars: any;
}): Promise<
  {
    role: "system" | "user" | "assistant";
    content: string;
  }[]
> => {
  return [
    {
      role: "system",
      content: dedent`You're a helpful assistant`,
    },
    {
      role: "user",
      content: dedent`Hello`,
    },
    {
      role: "assistant",
      content: dedent`Hello`,
    },
    {
      role: "user",
      content: dedent`${vars.question}`,
    },
  ];
};
I tried directly using the .ts file in the yaml but then it's not picking up the variables for some reason.
Copy code
yaml
description: "Test eval"
prompts: [dist/prompt.js]
providers:
  - id: azureopenai:chat:gpt3-16k
    config:
      apiHost: server-openai.openai.azure.com
      temperature: 0.4
      apiVersion: 2024-02-15-preview

tests:
  - vars:
      question: What's the weather like today?
    assert:
      - type: javascript
        value: file://test-file.js
I switched to the
0.49.3
and it worked (but I don't have the newest feature) so the issue is most likely with the latest version.
i
I think I see the problem, it looks like the function is now imported with
.default
due to the es6 export. There was a related change in this area. I should be able to publish a fix in the next few hours
@abhagsain is there any chance you can share
dist/prompt.js
so I can test the fix?
a
Sure, here's the response
Copy code
js
"use strict";
var __awaiter = (this && this.__awaiter) || function (thisArg, _arguments, P, generator) {
    function adopt(value) { return value instanceof P ? value : new P(function (resolve) { resolve(value); }); }
    return new (P || (P = Promise))(function (resolve, reject) {
        function fulfilled(value) { try { step(generator.next(value)); } catch (e) { reject(e); } }
        function rejected(value) { try { step(generator["throw"](value)); } catch (e) { reject(e); } }
        function step(result) { result.done ? resolve(result.value) : adopt(result.value).then(fulfilled, rejected); }
        step((generator = generator.apply(thisArg, _arguments || [])).next());
    });
};
var __importDefault = (this && this.__importDefault) || function (mod) {
    return (mod && mod.__esModule) ? mod : { "default": mod };
};
Object.defineProperty(exports, "__esModule", { value: true });
const dedent_1 = __importDefault(require("dedent"));
exports.default = (_a) => __awaiter(void 0, [_a], void 0, function* ({ vars: { noOfSlides, text, additionalInfo = "" }, }) {
    return [
        {
            role: "system",
            content: (0, dedent_1.default) `You're a helpful assistant`,
        },
    ];
});
i
thanks, that's really helpful. fix is here https://github.com/promptfoo/promptfoo/pull/613 The issue had to do with interoperability between es modules/commonjs and the way your build pipeline (and I'm sure other people's build pipelines too) wraps the module
a
Thank you, please let me know once this is released. For now I'll update the node_module file
i
@abhagsain it's released in 0.50.1. Please LMK if you run into issues
a
Yes, there's an issue. 😅 I was using
npx promptfoo@latest
and It's not picking up the path to the prompt file now. I'm always getting
Copy code
pf eval --verbose --no-cache
Cache is disabled.
You must provide at least 1 prompt
This is the YAML file. I switched to
0.50.0
it works but with the previous error and I changed the value in the node_modules and it's working fine.
Copy code
yaml
description: "Summarize texts "
prompts: [dist/summarize-prompt.js]
Also, did you change something here? Earlier, it used to show the transpiled js code, but now it shows classic [object Object]. And It would be great to see the temperature value in the UI. I'm testing different temperatures, but I can't see it in the UI. https://cdn.discordapp.com/attachments/1224455451706392597/1224769290494808134/image.png?ex=661eb249&is=660c3d49&hm=f4bbb42b693f306df3f7d882136c538547d0937ae38d47a38fa7e2ea30f94aa8&
i
That's strange, I tested it locally with the
dist/prompt.js
you provided 😱😱
to see temperature, easiest way for now is to use
label
. For example:
Copy code
yaml
providers:
  - id: azureopenai:chat:gpt3-16k
    label: 'gpt3-16k high temperature'
    config:
      apiHost: server-openai.openai.azure.com
      temperature: 0.4
      apiVersion: 2024-02-15-preview
a
is it working for you?
i
yeah, my full setup is this:
Copy code
yaml
prompts:
  - file://dist/prompt.js

providers:
  - openai:gpt-3.5-turbo-0613

tests:
  - vars:
      topic: foo
and then this file
and running
promptfoo eval
in the same directory as the config
a
dming you a video
25 Views