https://promptfoo.dev/ logo
Show RAG pipeline performance over multiple iterat...
# ❓|questions
g
Hi, i would like to be able to show my progress at the end of each sprint over the coming weeks of development of my RAG chatbot. During the next weeks i will probably have to update my index (vertex AI). One obvious solution would be to have multiple indicies running, to be able to do a vector-database or E2E comparison , however having all the old indicies live as well will be too expensive for this project. I am not sure if promptfoo is supposed to be used to show the incremental progress over a longer period of time. Any ideas on this would be highly appreciated. I hope to create a graph that looks something like the attached picture, being able to switch out the y-axis with different metrics from my evaluations. I can of course download the data from the interface, however i am wondering if someone already has a script to visualize the below data? https://cdn.discordapp.com/attachments/1224993914017157211/1224993914268811335/image.png?ex=661f837c&is=660d0e7c&hm=9f319bf54cbb2e96bca4c59226dd34d88093c11bf58811b0707a24b8fb56969d&
i
This is a great idea, just not something currently supported 🤔 At a minimum, seems like a table view of evals similar to the existing views for prompts/datasets would be helpful. Then at least you could easily copy the tabular data for pass rate
Are you running the evals with a single provider/prompt? So the output is a single column?
g
Thank you for the respose Ian! At the end of each cycle i would like to pick out the best performing provider, so yes - single column. I have reverse engineneered the data model from the downloaded JSON (which i am currently downloading manually), to extract relevant field eg. evaluation scores. This data is then stored in a tabular fashion for each cycle. The table also has the query attached, but redacted due to data privacy concerns. The graph would be nice to have in the browser view for quick insights on where to put development efforts when running in single column mode. https://cdn.discordapp.com/attachments/1224993914017157211/1225409417650901043/image.png?ex=66210674&is=660e9174&hm=2c16c87f480bea76fe3f17379d9fd19061fb2f0439f63556e4b3541a79173c48& https://cdn.discordapp.com/attachments/1224993914017157211/1225409417936375921/image.png?ex=66210674&is=660e9174&hm=a3707873d53bb462bb6b60e35f01b49ef3a45bd9d03f66340ee01b325865f0a3&
@ian - could you point me to the data model used for outputting the json file?
i
yep - here it is
g
Thanks! Your work is highly appreciated!
i
@gustavisaksen working on a simple view for this Priority would be just to make the info available for starters. Then can worry about nicer visuals 🙂 Let me know if you have any thoughts/feedback https://cdn.discordapp.com/attachments/1224993914017157211/1225440564355530752/image.png?ex=66212376&is=660eae76&hm=97200bcfd0e2a35976932fdd8ed4d11cf9906f6ed9029e09d2d4d1dcba46a4b3&
g
Thank you @User! I am amazed by your responsiveness, and help 🙂 Critical insights for me when developing / improving the RAG chatbot would be to see why my metrics are failing across a larger test-set. Here i would like to see the count / percentage of failures types eg. "similarity: 5/10 failures, factuailty 3/8 failures". This would give me insights on whether i should develop data foundation, retrieval engine, or prompt engineering. Maybe we could implement a dropdown allowing for percentage vs. count view? These columns would just be added to your existing table
Also an export button of the evaluation to csv would be highly appreciated.