https://promptfoo.dev/ logo
System Prompts of Unusual Size Not Playing Nicely ...
# ❓|questions
n
I am using promptfoo against azure's openai chatgpt4 API to test out some "role playing" prompts. The prompt file I wrote uses the _conversation variable to evaluate an ongoing conversation. I prefix the conversation with a system prompt to help orient the AIs persona. When I do this with a relatively terse system prompt (think a sentence or two), "promptfoo eval" runs, completes in 1-3 minutes, and returns the table output. The contents of the table output meet my expectations given the terse prompt. When i do this with an enormous system prompt (think the contents of a 10 page guide used to instruct human participants how to perform during the role playing session ... about 6k tokens) "promptfoo eval" grinds to an absolute halt ... the node process churns and churns and churns. I have left it run for more than an hour before and have seen promptfoo output the table output multiple times (i am not sure if it is the exact same table output printed to the console multiple times, or it was the exact same output printed multiple times) with the node process still not exiting. When i run promptfoo as "promptfoo eval --no-table", promptfoo runs, completes in 1-3 minutes, and the contents of output.json shows the back and forth with the azure endpoint with satisfactory generated completions. So there appears to be something happening when displaying the table output with a giant ass system prompt. Has anyone run into anything like this with promptfoo? running: - promptfoo 0.33.1 - node 16.14.0 - macos Sonoma 14.2
While writing this, I realized that I am using an old and crusty version of node. I updated to the latest LTS (v20.10.0) and the problematic behavior is almost identical as when run with node v16.14.0.
i have been playing with this more ... and i think it is just a result of how the CLI table is constructed. the vars and responses in the table are limited to a max of 250 characters by default. there is no such limit on the prompts displayed in the header. why a 4000k header causes such an issue ... no idea.
i
Yeah, I think you are right. The table rendering library that we use is just very inefficient. I agree that the header should be truncated, I'll get in a fix for this 🙂 Nice catch
fix here:
n
@ian you are a rockstar!