https://promptfoo.dev/ logo
Running any Eval in web results in Syntax Error <s...
# ❓|questions
e
Hi folks, I'm new to promptfoo and I'm trying it out for the first time, but instead of running the cli/quick-start I went straigth to the web server. I configured a docker-compose running the latest image (version 0.109.1) and then I tried some basic examples from the quick start and from the examples repository folder. All Evals attemps raises a similar error message: > SyntaxError: Unexpected token 'I', "I don't ha"... is not valid JSON > > SyntaxError: Unexpected token 'I', "I don't ha"... is not valid JSON > at JSON.parse () > at eval (eval at getInlineTransformFunction (/app/src/util/transform.ts:97:10), :3:13) > at transform (/app/src/util/transform.ts:159:37) > at runAssertion (/app/src/assertions/index.ts:131:14) > at /app/src/assertions/index.ts:363:22 I have tried with google and openrouter providers and results are always the same. Given the errors output it seems that the Evals are calling the providers API, but the response does not seems to be a valid JSON. Instead it seems that the web server is trying to parse the actual answer (the LLM response) created by the provider. Some examples are: > SyntaxError: Unexpected token 'I', "I don't ha"... is not valid JSON > SyntaxError: Unexpected token 'I', "I'm sorry,"... is not valid JSON > SyntaxError: Unexpected token 'H', "Here are a"... is not valid JSON > SyntaxError: Unexpected token 'A', "Ahoy, mate"... is not valid JSON Am I missing some configuration? Any ideas on what I'm doing wrong? https://cdn.discordapp.com/attachments/1360053293484605622/1360053293673484338/image.png?ex=67f9b7d5&is=67f86655&hm=278e125e99fd9bd76ed6c891f1d64097b7f572af7029ab0ee20ab06c24efb6a7&
m
Hey @ericchaves! Thanks for flagging this. Looks like you're getting a refusal which is breaking JSON parsing. This is something I would like to fix for you and other promptfoo users. Could you share your full promptfooconfig here? That way, I can take a closer look at what's causing the issue. Alternatively, if it's easier, feel free to book some time on my calendar, and we can debug it together: https://cal.com/michael-dangelo/30min
e
@michaelmichaelmichael sure! I'm attaching the following files: - promptfoo-demo.zip: my project scruture used to run the promptfoo using docker composer minus my .env file whre the env for provider keys are declared (GOOGLE_API_KEY and OPENROUTER_API_KEY only) - promptfoo.db: the sqlite3 from inside running container - prompt-*.yaml: one of the YAML evals that I used which generate the mentioned error. Although any eval produces the same error. https://cdn.discordapp.com/attachments/1360053293484605622/1360230066050830558/promptfoo-error.yaml?ex=67fa5c77&is=67f90af7&hm=71c0c6fa59faaa3347a7046fa31149747db5cca0458ab1c2537432eb2377f1ce& https://cdn.discordapp.com/attachments/1360053293484605622/1360230066403016774/promptfoo.db?ex=67fa5c77&is=67f90af7&hm=033ff850e5e680c25114177d5b698c7ad2f78a6ebcdecc25ea1e630f6c3c2db5& https://cdn.discordapp.com/attachments/1360053293484605622/1360230066751017031/promptfoo-demo.zip?ex=67fa5c77&is=67f90af7&hm=f15b69b91c44224a0e9b83358210b1dc10f28b1b1af52616ed7c56ada31d360f&
please let me know if you need anythng else.
Hey @michaelmichaelmichael jut checking if there is anything else on my side that may help with this issue. Best regards.
Hi folks. after some digging around I figured what's happening and foud the error source. For future users, when using the server/web view, the YAML provided into the UI seems to be merged with a local eval config state stored in localstorage. In my case the initial state had a defaultTestCase declaration with a transformation that expected the LLM output to be a JSON, but my current promptfoo YAML was just calling LLM and expcting return in plain text natural language, therefore the transformation was breaking. All I had to do was to delete the localstorage and now the evals are running smoothly!! @michaelmichaelmichael did I got it right?
n
@michaelmichaelmichael ...