https://promptfoo.dev/ logo
JavaScript assertions not working
# ❓|questions
j
Hi Promptfoo! I'm loving the product but I've been troubleshooting why my JavaScript assertions haven't been working for a couple hours and maybe I'm missing something obvious I am testing an OpenAI Assistant that has 4 possible function calls configured. In the docs here "https://www.promptfoo.dev/docs/configuration/expected-outputs/javascript/" it says "If the LLM outputs a JSON object (such as in the case of tool/function calls), then output will already be parsed as an object", however I have not found this to be the case Regardless of whether I set my provider's response_format to text or to json I always get test results that look like this: question: What is the exchange rate of 2 eth to dai on base? User wallet address is 0x59594B66Ef90e2aF2BEC5A49E81CeAE42c8ad556 outputs: is-valid-openai-tools-call
Copy code
OpenAI did not return a valid-looking tools response: "[Call function exchangeRate with arguments {\"arguments\":[{\"fromToken\":\"ETH\",\"toToken\":\"DAI\",\"fromAmount\":\"2\"}]}]\n\n[Function output: null]"

[Call function exchangeRate with arguments {"arguments":[{"fromToken":"ETH","toToken":"DAI","fromAmount":"2"}]}]

[Function output: null]
is-json
Copy code
Expected output to be valid JSON

OpenAI did not return a valid-looking tools response: "[Call function exchangeRate with arguments {\"arguments\":[{\"fromToken\":\"ETH\",\"toToken\":\"DAI\",\"fromAmount\":\"2\"}]}]\n\n[Function output: null]"
Could it be that it fails to trigger the JS callbacks and that is why the function output appears as null? Shouldn't the default case be that it is always parseable though? for reference my promptfoo config looks like this this is my promptfooconfig.yaml which is generated programatically
Copy code
prompts:
  - '{{question}}'
defaultTest:
  options:
    providers:
      text:
        id: openai:gpt-4o
      embedding:
        id: openai:embeddings:text-embedding-ada-002
providers:
  - id: openai:assistant:asst_REDACTED_ID_HERE
    config:
      model: gpt-4o
      temperature: 1
      top_p: 1
      response_format: json_object
      instructions: file://base-prompt.txt
      tools:
        - type: function
          function:
            name: exchangeRate
            description: >-
              Description text redacted
            parameters:
              type: object
              properties:
                arguments:
                  type: array
                  items:
                    type: object
                    properties:
                      fromToken:
                        type: string
                      toToken:
                        type: string
                      fromAmount:
                        type: string
                      toAmount:
                        type: string
              required:
                - arguments
      messages:
        - role: system
          content: file://base-prompt.txt
        - role: user
          content: '{{question}}'
tests:
  - vars:
      question: >-
        What is the exchange rate of 2 eth to dai on base? User wallet address
        is 0x59594B66Ef90e2aF2BEC5A49E81CeAE42c8ad556
    assert:
      - type: is-json
        value: true
      - type: is-valid-openai-tools-call
        value: true
Is it correct that if I use functionToolCallbacks then I would not be able to use the is-json and is-valid-openai-tools-call helpers since the callback takes in text and returns a Promise? In this scenario how would you mock the output while still being able to use those json helpers?