https://promptfoo.dev/ logo
functions & python
# ❓|questions
b
I've got two issues: When I try to evaluate a prompt that uses the function calling of open Ai, I Always get [object Object] as my output, even with the example of the documentation. I have the intuition that it Can be resolved easily but I can't figure it out. When I want to use a python script for a custom assertion, I get an error saying the assertion must return boolean or a {pass, score, reason} structure, but that's basically what my script return (the {pass, score, reason}) and even with the python example in the Doc, I got the same issue.
i
could you show me an example config? that way I can try on my end and it might be easier to troubleshoot 🙂
b
here is a config with the example of the doc:
Copy code
providers:
  - id: openai:gpt-3.5-turbo-0613
    config:
      temperature: 0.1
      functions: [
        {
            "name": "get_current_weather",
            "description": "Get the current weather in a given location",
            "parameters": {
              "type": "object",
              "properties": {
                "location": {
                  "type": "string",
                  "description": "The city and state, e.g. San Francisco, CA"
                },
                "unit": {
                  "type": "string",
                  "enum": ["celsius", "fahrenheit"]
                }
              },
              "required": ["location"]
            }
          }
      ]
prompts: [prompt.json]
tests:
    - description : "Test if json"
      vars: 
        INPUT: Boston
      assert:
        - type: is-json
This is how I configured the "functions" in my case. The promt is this :
Copy code
[
  {
    "role": "user",
    "content": "What is the weather like in {{INPUT}}?"
  }
]
And the output is [object Object]
i
As a workaround, you can set
options.postprocess
on the test case:
Copy code
yaml
tests:
    - description : "Test if json"
      options:
        postprocess: JSON.stringify(output)
      vars:
        INPUT: Boston
      assert:
        - type: is-json
I'll make sure a better fix lands
b
Thank you for the answer !