I am using promptfoo against azure's openai chatgpt4 API to test out some "role playing" prompts. The prompt file I wrote uses the _conversation variable to evaluate an ongoing conversation. I prefix the conversation with a system prompt to help orient the AIs persona.
When I do this with a relatively terse system prompt (think a sentence or two), "promptfoo eval" runs, completes in 1-3 minutes, and returns the table output. The contents of the table output meet my expectations given the terse prompt.
When i do this with an enormous system prompt (think the contents of a 10 page guide used to instruct human participants how to perform during the role playing session ... about 6k tokens) "promptfoo eval" grinds to an absolute halt ... the node process churns and churns and churns. I have left it run for more than an hour before and have seen promptfoo output the table output multiple times (i am not sure if it is the exact same table output printed to the console multiple times, or it was the exact same output printed multiple times) with the node process still not exiting.
When i run promptfoo as "promptfoo eval --no-table", promptfoo runs, completes in 1-3 minutes, and the contents of output.json shows the back and forth with the azure endpoint with satisfactory generated completions.
So there appears to be something happening when displaying the table output with a giant ass system prompt.
Has anyone run into anything like this with promptfoo?
running:
- promptfoo 0.33.1
- node 16.14.0
- macos Sonoma 14.2