https://promptfoo.dev/ logo
Introduce a BigQuery Adapter to Enable Custom Inpu...
# ❓|questions
p
I'm interested in gathering the team's feedback on the potential introduction of a BigQuery adapter as a feature in the framework. I am exploring the use case of running automated daily evaluations with your framework. Since, we want to run this daily, it could be more efficient to use input and output queries instead of creating a CSV file. The framework would use these queries to retrieve the input dataset and direct the output. A potential approach could look something like - 1. Incorporate the BigQuery configuration into the
promptfooconfig.yaml.
2. Store the input & output query in
.sql
file, link it in the
promptfooconfig.yaml
3. Use these configuration in the
evaluate()
function and CLI. 4. Modify
evaluate
to store the result in big query Looking forward to hearing your thoughts and suggestions.
a
Adding myself to this conversation
@ian we are currently discussing internally on how to best use promptfoo for running evaluations on new datasets that we get on a daily basis that lives in bigquery, and be able to store the output to bigquery as well
i
👀
Seems like a useful idea. Here is where tests are currently read and parsed We'd have to decide the configuration structure, add the BQ client, but should be relatively straightforward to read
Likewise, here is where results are written:
a
we also plan to write the data to a bigquery table (sink) as well
i
got it. do you think you'll call the library direct from js or use the cli?
a
so this change combine with the PR I made previously will unlock the ability to run evaluations on already existing output data, with data that is not directly available locally
we are not sure about this yet...
but we do want to make it so that other DS/MLEs that are comfortable with Python to be able to use it as well
i
makes sense
p
My preference for when we introduce this feature, it should be with the cli and then probably add it to js. Cli seems to be easiest to start using for other DS/MLEs.
I am going to try and give it a go. I will start with the interface and once we agree on that, can start the integration!
i
That sounds great. I'm happy to discuss anytime if you need a pointer or just want to bounce ideas
p
Thanks you 🙂
a
@pragyadaga I am gonna start writing a proposal on the architecture and how people can use it, its gonna be useful for our conversation here and I think we gonna need to share that internally as well to different teams