https://promptfoo.dev/ logo
Correct way to test a ReAct LangGraph agent
# ❓|questions
w
Hi there! Just want to check if I'm going about this the right way. Is promptfoo the right tool to be using for testing my ReAct agent? I was planning to write a simple javascript (typescript) provider to invoke my langgraph app. (Just a basic ReAct agent. 2 tools, multiple tool turn calls before providing a text response) I only really care about the graph's final output message, as well as a property on the state, and I do really want to use promptfoo's cool features like llm-rubric assertion's to evaluate the message, but I don't think I'd be using the rest of this framework properly if my prompt is hardcoded outside of the yaml file? What would be the best way to structure these tests? My current prompt is a system + user message, written using langchain's ChatPromptTe plate class
Copy code
ts
export default ChatPromptTemplate.fromMessages([
    ['system', systemMessageText],
    new MessagesPlaceholder('messages')
]);
After reading into this, should I perhaps adapt my langgraph graph to support re-building the agent, in order to allow it to be ran with different prompts? Or should I just not use the input prompts at all? Or, considering this is a ReAct agent with just 2 tools, should I instead try to replicate it with the regular providers and tool definitions? Is it possible to have multiple tool turns that way?
t
@Waz sorry it's taken a while to get to this - do you still need help here or are you sorted?
w
Oh no worries I ended up writing a custom provider to run the graph. Seems to work well. Is this the correct way to do it? So far it's working good, especially after adding token count and cost information, though my one other question is about recording tool calls and writing assertions around it. I opened another post for it, though now I'm wondering if that is out of scope of this framework
t
That's good! Appreciate the update. I'm looking into the other one so I will tag someone there