https://promptfoo.dev/ logo
Does promptfoo supprot custom agents testing??
# ❓|questions
s
I have custom agents built using Theia AI. Is promptfoo a right framework to evaluate custom agents? Can i handle this using custom typescript/python script?
t
Hi @saraswathirekhala_44593! Sure - have you had a look here? https://www.promptfoo.dev/docs/providers/custom-script/ https://www.promptfoo.dev/docs/providers/custom-api/ https://www.promptfoo.dev/docs/providers/python/ There's more for providers that are webhook-based, websocket-based, or provide a http endpoint to test (in the table at https://www.promptfoo.dev/docs/providers/ )
s
Yes @Tabs [Promptfoo] .... I have checked this custom implementation. Similar to direct MCP support we donot have any agent support in promptfoo.. everything should be handled from scratch?
t
If there's no framework or language already supported and you want to use Promptfoo to test it then yes you'll need to write that part yourself
i
+1 the best way would be to use a [typescript/javascript provider](https://www.promptfoo.dev/docs/providers/custom-api/) with theia ai