jahabeebs
02/19/2025, 6:25 PMOpenAI did not return a valid-looking tools response: "[Call function exchangeRate with arguments {\"arguments\":[{\"fromToken\":\"ETH\",\"toToken\":\"DAI\",\"fromAmount\":\"2\"}]}]\n\n[Function output: null]"
[Call function exchangeRate with arguments {"arguments":[{"fromToken":"ETH","toToken":"DAI","fromAmount":"2"}]}]
[Function output: null]
is-json
Expected output to be valid JSON
OpenAI did not return a valid-looking tools response: "[Call function exchangeRate with arguments {\"arguments\":[{\"fromToken\":\"ETH\",\"toToken\":\"DAI\",\"fromAmount\":\"2\"}]}]\n\n[Function output: null]"
Could it be that it fails to trigger the JS callbacks and that is why the function output appears as null? Shouldn't the default case be that it is always parseable though? for reference my promptfoo config looks like this
this is my promptfooconfig.yaml which is generated programatically
prompts:
- '{{question}}'
defaultTest:
options:
providers:
text:
id: openai:gpt-4o
embedding:
id: openai:embeddings:text-embedding-ada-002
providers:
- id: openai:assistant:asst_REDACTED_ID_HERE
config:
model: gpt-4o
temperature: 1
top_p: 1
response_format: json_object
instructions: file://base-prompt.txt
tools:
- type: function
function:
name: exchangeRate
description: >-
Description text redacted
parameters:
type: object
properties:
arguments:
type: array
items:
type: object
properties:
fromToken:
type: string
toToken:
type: string
fromAmount:
type: string
toAmount:
type: string
required:
- arguments
messages:
- role: system
content: file://base-prompt.txt
- role: user
content: '{{question}}'
tests:
- vars:
question: >-
What is the exchange rate of 2 eth to dai on base? User wallet address
is 0x59594B66Ef90e2aF2BEC5A49E81CeAE42c8ad556
assert:
- type: is-json
value: true
- type: is-valid-openai-tools-call
value: true
Is it correct that if I use functionToolCallbacks then I would not be able to use the is-json and is-valid-openai-tools-call helpers since the callback takes in text and returns a Promise? In this scenario how would you mock the output while still being able to use those json helpers?