I have a stateful agent (data analysis agent that has access to a coding sandbox with a dataframe) that I want to evaluate with the simulated user provider. The issue is that every call_api is stateless and we are expected to feed the entire conversation/state as a string each time. Is there a recommended way to keep the same agent/client around for a few calls?

I'd recommend storing the conversation history server-side and then returning a session id when the conversation is initialized. See server-side management https://www.promptfoo.dev/docs/providers/http/#session-management

is there any way to keep the same process if you're using simulated user? that way we can keep an agent in memory and it gets cleaned up when the eval is done. if I take the session management approach I need to start a persistent stateful server before running any promptfoo evals and it won't be cleaned up until all the evals run.

Are you using a custom python provider to interface with your stateful agent?

yes

providers:
  - file://promptfoo_utils/pf_provider.py

We have a fix for 5554 in progress right now. I will follow up here once that ships – optimistically this week.