https://promptfoo.dev/ logo
Assertion that calls a scoring API
# ❓|questions
s
Hi, new to promptfoo here. The platform looks great btw :). Had a question about the best way to add an assertion integration. I'm looking to integrate a LLM-as-a-judge alternative my team has been working on- called the "pi scorer". We'd like to include it as an assertion in promptfoo. This scorer uses an encoder head to provide faster and deterministic scores. Like an LLM, it can handle any text subject and criteria. You can learn more here: https://docs.withpi.ai https://build.withpi.ai We were wondering if we could add an assertion, rather than integrating with the existing python assertion - so that people can discover this alternative scoring method. A few questions; 1. Can we add this as an official assertion, so that people don't need to write the python code to call this scorer? 2. This assertion requires using an API key. How would testing work in this case? Should we mock calls to our api so that the api key isn't needed during tests? Thanks for your time. Looking forward to hearing from y'all.