https://promptfoo.dev/ logo
JSON errors during prompt generation
# ❓|questions
b
When I'm attempting to generate my redteam prompts (promptfoo redteam generate), I keep consistently running into "SyntaxError" issues in JSON, seemingly at inconsistent points during generation. I've tried multiple times and keep running into the same error:
Copy code
[chat.js:161]     completions API response: {"id":"chatcmpl-BfWK1RSvr3LAckI7hoUHM9dYo0Zce","object":"chat.completion","created":1749235393,"model":"gpt-4o-mini-2024-07-18","choices":[{"index":0,"message":{}
<anonymous_script>:430


SyntaxError: Expected ',' or '}' after property value in JSON at position 1966 (line 430 column 1)
    at JSON.parse (<anonymous>)
    at encodeMathPrompt (/opt/homebrew/Cellar/promptfoo/0.114.5/libexec/lib/node_modules/promptfoo/dist/src/redteam/strategies/mathPrompt.js:95:32)
    at process.processTicksAndRejections (node:internal/process/task_queues:105:5)
    at async addMathPrompt (/opt/homebrew/Cellar/promptfoo/0.114.5/libexec/lib/node_modules/promptfoo/dist/src/redteam/strategies/mathPrompt.js:122:33)
    at async action (/opt/homebrew/Cellar/promptfoo/0.114.5/libexec/lib/node_modules/promptfoo/dist/src/redteam/strategies/index.js:195:34)
    at async applyStrategies (/opt/homebrew/Cellar/promptfoo/0.114.5/libexec/lib/node_modules/promptfoo/dist/src/redteam/index.js:241:35)
    at async synthesize (/opt/homebrew/Cellar/promptfoo/0.114.5/libexec/lib/node_modules/promptfoo/dist/src/redteam/index.js:678:85)
    at async doGenerateRedteam (/opt/homebrew/Cellar/promptfoo/0.114.5/libexec/lib/node_modules/promptfoo/dist/src/redteam/commands/generate.js:243:88)
Is this a Promptfoo bug by chance? Or is it possible I'm doing something wrong? Happy to DM over my promptfooconfig.yaml if helpful
An update on this, it looks like it was pretty consistently failing towards the end of the generation in the math-prompt strategy, fwiw
i
hmm, have you overridden the redteam provider by any chance?
this should help. will be in the next release https://github.com/promptfoo/promptfoo/pull/4361