Hi folks, I'm new to promptfoo and I'm trying it out for the first time, but instead of running the cli/quick-start I went straigth to the web server.
I configured a docker-compose running the latest image (version 0.109.1) and then I tried some basic examples from the quick start and from the examples repository folder.
All Evals attemps raises a similar error message:
> SyntaxError: Unexpected token 'I', "I don't ha"... is not valid JSON
>
> SyntaxError: Unexpected token 'I', "I don't ha"... is not valid JSON
> at JSON.parse (
)
> at eval (eval at getInlineTransformFunction (/app/src/util/transform.ts:97:10),
:3:13)
> at transform (/app/src/util/transform.ts:159:37)
> at runAssertion (/app/src/assertions/index.ts:131:14)
> at /app/src/assertions/index.ts:363:22
I have tried with google and openrouter providers and results are always the same.
Given the errors output it seems that the Evals are calling the providers API, but the response does not seems to be a valid JSON. Instead it seems that the web server is trying to parse the actual answer (the LLM response) created by the provider. Some examples are:
> SyntaxError: Unexpected token 'I', "I don't ha"... is not valid JSON
> SyntaxError: Unexpected token 'I', "I'm sorry,"... is not valid JSON
> SyntaxError: Unexpected token 'H', "Here are a"... is not valid JSON
> SyntaxError: Unexpected token 'A', "Ahoy, mate"... is not valid JSON
Am I missing some configuration? Any ideas on what I'm doing wrong?
https://cdn.discordapp.com/attachments/1360053293484605622/1360053293673484338/image.png?ex=67f9b7d5&is=67f86655&hm=278e125e99fd9bd76ed6c891f1d64097b7f572af7029ab0ee20ab06c24efb6a7&