blank2011
12/19/2023, 8:02 AMian
12/21/2023, 5:00 AMblank2011
12/21/2023, 6:57 AMblank2011
01/02/2024, 11:41 AMicakirlar
01/03/2024, 1:30 PMian
01/03/2024, 5:48 PMpromptfoo eval results can be run locally in CI, but you also want to self-host the web viewer.
- I think the best approach here would be similar to the existing "share" functionality, except it would send the eval results to the specified host.
- The sharing host itself is just a webui + kv store running in a docker container.
Does that sound good to you all @blank2011 @icakirlar? If so, I expected to land this change within the next week or so.blank2011
01/04/2024, 7:27 AMprompts: [genactionables.txt]
providers:
- "exec:python3 generate_actionables.py"
defaultTest:
options:
provider:
id: azureopenai:embeddings:text-embedding-ada-002
config:
apiHost: "${OPENAI_API_BASE_V2}"
apiKey: ${OPENAI_API_KEY}
tests: genactionables.csv
Then for continous deployment, when we had new changes made to codebase, let's say we added new test cases. Then the new commits would trigger the pipeline. Which will build a docker image.blank2011
01/04/2024, 7:27 AM# Start with a base Ubuntu 22.04 image
FROM ubuntu:22.04
# Set environment variables to non-interactive (this is for automated build processes)
ENV DEBIAN_FRONTEND=noninteractive
ENV TZ=Asia/Kolkata
# Create app directory (assuming your app will reside there)
WORKDIR /usr/src/app
# Install Node.js, Python, pip, and other dependencies
RUN apt-get update && apt-get install -y \
curl \
python3.11 \
python3-pip \
python3-dev \
gettext-base \
&& curl -fsSL https://deb.nodesource.com/setup_18.x | bash - \
&& apt-get install -y nodejs \
&& npm install -g npm@latest \
&& npm install -g promptfoo \
&& rm -rf /var/lib/apt/lists/*
# Copy your Node app source
COPY . /usr/src/app/
# Set environment variables for Python
ENV PYTHONDONTWRITEBYTECODE 1
ENV PYTHONUNBUFFERED 1
# Install pip dependencies
RUN pip3 install --no-cache-dir --upgrade pip \
&& pip3 install --no-cache-dir -r requirements.txt
# Expose the port your app runs on
EXPOSE 80
# Make the entrypoint script executable
RUN chmod +x /usr/src/app/entrypoint.sh
# Set the entrypoint script
ENTRYPOINT ["sh", "/usr/src/app/entrypoint.sh"]blank2011
01/04/2024, 7:28 AM#!/bin/bash
# # Source the .env file if it exists
# if [ -f /usr/src/app/.env ]; then
# export $(grep -v '^#' /usr/src/app/.env | xargs)
# fi
## setup symlink to promptfoo
ln -s /root/.promptfoo /promptfoo
# Substitute environment variables in the YAML templates and output to the final config files
envsubst < /usr/src/app/promptfoo/chain_step_1/genactionconfig.yaml.template > /usr/src/app/promptfoo/chain_step_1/genactionconfig.yaml &&
envsubst < /usr/src/app/promptfoo/chain_step_2/OPconfig.yaml.template > /usr/src/app/promptfoo/chain_step_2/OPconfig.yaml &&
envsubst < /usr/src/app/promptfoo/chain_step_3/specificationsConfig.yaml.template > /usr/src/app/promptfoo/chain_step_3/specificationsConfig.yaml
# Set the default application port
APP_PORT=${PORT:-80}
# Run promptfoo evals for each configuration file
npx promptfoo@latest eval -c /usr/src/app/promptfoo/chain_step_1/genactionconfig.yaml
npx promptfoo@latest eval -c /usr/src/app/promptfoo/chain_step_2/OPconfig.yaml
npx promptfoo@latest eval -c /usr/src/app/promptfoo/chain_step_3/specificationsConfig.yaml
# Run the promptfoo view
npx promptfoo@latest view --yes --port $APP_PORTblank2011
01/04/2024, 7:29 AM/promptfooblank2011
01/04/2024, 7:29 AMblank2011
01/04/2024, 7:33 AMicakirlar
01/04/2024, 9:09 AMicakirlar
01/15/2024, 10:27 AMian
01/17/2024, 5:11 PMteddy06894
01/22/2024, 10:09 PMian
01/24/2024, 1:57 AMpromptfoo share server. Supported stores are memory, filesystem, and redis.teddy06894
01/24/2024, 6:58 AMian
01/24/2024, 7:04 AMpromptfoo share for CLI/node package evalsian
01/24/2024, 7:04 AMteddy06894
01/24/2024, 7:18 AMteddy06894
01/25/2024, 6:50 AMian
01/25/2024, 9:45 PMteddy06894
01/25/2024, 10:33 PMteddy06894
01/28/2024, 12:18 PMian
01/28/2024, 4:22 PMteddy06894
02/02/2024, 7:00 AMian
02/06/2024, 6:22 AMteddy06894
02/06/2024, 9:36 PMteddy06894
02/06/2024, 9:36 PMblank2011
02/09/2024, 6:34 AM