https://promptfoo.dev/ logo
Could not extract JSON from LLm rubric error while...
# ❓|questions
d
I am using the customProvider.js and using ollama:llama3.1 as my provider and sometimes I get this error, I've logged the output I receive from my custom provider and it works fine but I am unable to understand this particular error Here is promptfooconfig.yaml file for reference description: 'RAG - End to end test with comparison' prompts: - prompts.json providers: - id: customProvider.js tests: - vars: filePath: Question - What is the difference between supervised and unsupervised learning in machine learning noOfvariants: 1 application: AGA assert: - type: llm-rubric value: 'evaluate how good the answer is' provider: ollama:llama3.1
i
I'm guessing llama does not output well-formed JSON in these cases. That's why it's happening intermittently
You could try a larger llama model, if you machine supports it. That will probably make it output JSON more reliably. Alternatively, you could figure out a way to apply JSON guardrails with Ollama