The next version of Drupal called Drupal CMS will be shipping with the possibility of enabling AI and AI Agents out of the box. The goal is that this will become the defacto standard version of Drupal that powers 10% of the top 10, 000 sites and allows a lot of ai application building to be verified and built through a UI.
We've had people recommend promptfoo is this the kind of things that would benefit from a closer integration with Drupal or something we could occasionally use or be inspired by?
One issue we have with evaluation frameworks is that we need to evaluate complete systems as if someone asked to add a field to an entity using a chatbot, its a mixture of chat history, an assistant, a bunch of agents, tools all working together and the whole thing needs to be evaluated ideally not just single prompts.
Is this a good place to start?