https://promptfoo.dev/ logo
Is it possible to share promptfoo eval results in ...
# ❓|questions
b
- For my usecase, my team needs to have the promptfoo eval results hosted in our URL/remote server instance. Currently, I have seen that there is a share command that creates a temporary share URL. And the data is stored in Supabase, which we don't own. - It would have been great if we could run promptfoo eval during a docker build and deploy that container in an Azure Web App, with the promptfoo web viewer running. Or results data could ideally be another database provider.
i
thanks for the feedback. I am actively working on a standalone docker container that can serve as the eval server!
b
Oh cool, I am looking forward to testing it 👍
@ian Just a query. When could I expect this implementation? In the next release?
i
I am also interested in this feature.
i
Just to make sure I understand and am building the right thing: -
promptfoo eval
results can be run locally in CI, but you also want to self-host the web viewer. - I think the best approach here would be similar to the existing "share" functionality, except it would send the eval results to the specified host. - The sharing host itself is just a webui + kv store running in a docker container. Does that sound good to you all @blank2011 @icakirlar? If so, I expected to land this change within the next week or so.
b
Hey @ian. For our team usecase, we don't want local promptfoo eval to be shown in self-hosted web viewer. We want it to run in a CI/CD pipeline( in our case its azure devOps pipeline). I actually went ahead and implemented for my usecase, since our team needed a demo for results asap. This is what I did. So I had Promptfoo Eval repo. Where there are prompt.txt, tests.csv, promptfooconfig.yaml.template promptfooconfig.yaml.template
Copy code
prompts: [genactionables.txt]
providers:
  - "exec:python3 generate_actionables.py"

defaultTest:
  options:
    provider:
      id: azureopenai:embeddings:text-embedding-ada-002
      config:
        apiHost: "${OPENAI_API_BASE_V2}"
        apiKey: ${OPENAI_API_KEY}

tests: genactionables.csv
Then for continous deployment, when we had new changes made to codebase, let's say we added new test cases. Then the new commits would trigger the pipeline. Which will build a docker image.
Dockerfile
Copy code
# Start with a base Ubuntu 22.04 image
FROM ubuntu:22.04


# Set environment variables to non-interactive (this is for automated build processes)
ENV DEBIAN_FRONTEND=noninteractive
ENV TZ=Asia/Kolkata

# Create app directory (assuming your app will reside there)
WORKDIR /usr/src/app

# Install Node.js, Python, pip, and other dependencies
RUN apt-get update && apt-get install -y \
    curl \
    python3.11 \
    python3-pip \
    python3-dev \
    gettext-base \
    && curl -fsSL https://deb.nodesource.com/setup_18.x | bash - \
    && apt-get install -y nodejs \
    && npm install -g npm@latest \
    && npm install -g promptfoo \
    && rm -rf /var/lib/apt/lists/*

# Copy your Node app source
COPY . /usr/src/app/

# Set environment variables for Python
ENV PYTHONDONTWRITEBYTECODE 1
ENV PYTHONUNBUFFERED 1

# Install pip dependencies
RUN pip3 install --no-cache-dir --upgrade pip \
    && pip3 install --no-cache-dir -r requirements.txt

# Expose the port your app runs on
EXPOSE 80

# Make the entrypoint script executable
RUN chmod +x /usr/src/app/entrypoint.sh

# Set the entrypoint script
ENTRYPOINT ["sh", "/usr/src/app/entrypoint.sh"]
entrypoint.sh
Copy code
#!/bin/bash

# # Source the .env file if it exists
# if [ -f /usr/src/app/.env ]; then
#     export $(grep -v '^#' /usr/src/app/.env | xargs)
# fi

## setup symlink to promptfoo
ln -s /root/.promptfoo /promptfoo

# Substitute environment variables in the YAML templates and output to the final config files
envsubst < /usr/src/app/promptfoo/chain_step_1/genactionconfig.yaml.template > /usr/src/app/promptfoo/chain_step_1/genactionconfig.yaml &&
envsubst < /usr/src/app/promptfoo/chain_step_2/OPconfig.yaml.template > /usr/src/app/promptfoo/chain_step_2/OPconfig.yaml &&
envsubst < /usr/src/app/promptfoo/chain_step_3/specificationsConfig.yaml.template > /usr/src/app/promptfoo/chain_step_3/specificationsConfig.yaml

# Set the default application port
APP_PORT=${PORT:-80}

# Run promptfoo evals for each configuration file
npx promptfoo@latest eval -c /usr/src/app/promptfoo/chain_step_1/genactionconfig.yaml
npx promptfoo@latest eval -c /usr/src/app/promptfoo/chain_step_2/OPconfig.yaml
npx promptfoo@latest eval -c /usr/src/app/promptfoo/chain_step_3/specificationsConfig.yaml

# Run the promptfoo view
npx promptfoo@latest view --yes --port $APP_PORT
ln -s /root/.promptfoo /promptfoo I also setup a azure fileshare, so the old promptfoo output won't be lost. Which is mounted at
/promptfoo
So that is how it is currently setup for us.
So if there are new changes made and committed, only then eval will be ran and the results will be shown in the deployed Azure web app.
i
@ian Your description is exactly what we are looking for. Thanks a lot! Looking forward to use it.
@ian Do you have any updates on this? I can try contributing to your effort here if you need any help.
i
Hey @icakirlar, still planning on this but took a little more time than expected. Aiming for next week. I've been meaning to get your thoughts on the KV store. Do you care about using something like redis, or would filesystem on Docker work?
t
Hey @ian any updates?
i
glad to report this is code complete and landing soon 🙂 It will be in the next release https://github.com/promptfoo/promptfoo/pull/408 This PR adds support for self-hosting the
promptfoo share
server. Supported stores are memory, filesystem, and redis.
t
Hi! But with that is possible to run evals directly using the UI self-hosted?
i
not with this change. This enables
promptfoo share
for CLI/node package evals
Adding support for the UI is a small step afterwards though
t
It would be very nice 🤘
i
@icakirlar The self-hosted
promptfoo share
functionality is included in release 0.37.0. It is documented here:
t
Do you have any plans to develop it?
i
yep!
t
Perfect, thanks!
ETA more or less?
i
this week most likely
t
Any news? 🫣
i
This turned out to be a little harder than expected. Still working on it
t
Oh 😞
Thanks
b
Hey @ian , So I had setup promptfoo in a Azure Web App container along with Azure FileShare storage(mentioned in the thread prior). And this is the gist of the part of the architecture. - So in the azure fileshare, the evals gets uploaded from local machine. - After it gets uploaded, the promptfoo dashboard shows the new evals in 5mins window(cronjob). So there is no eval running on the Web App, since we had budget on instances we could use. This setup has been working for my entire team effectively for over a month and the experience using promptfoo is great. I just wanted to put this idea out here, hoping this helps for others as well as an alternative.
2 Views