https://promptfoo.dev/ logo
Googlesheets eval with OpenAI chat API returning [...
# ❓|questions
r
I have an eval that uses a google sheet. The input and output are chat completion objects. In the screenshot you can see I don't actually get the output just the JS object type.
i
are you using one of the builtin
openai:
providers?
r
yes
Copy code
prompts: [prompts.json]
providers: 
  - id: openai:gpt-4
...
is using the built in causing this?
i
maybe a bug that I will look into fixing tomorrow. In the meantime you should be able to work around it by using a [postprocess directive](https://promptfoo.dev/docs/configuration/guide/#postprocessing):
Copy code
yaml
postprocess: JSON.stringify(output)
just to make sure I understand, this happens when the LLM output is valid JSON?
r
Yes for background I'm checking to see if the output matches the entire message. With function calls it's not enough to just check the message content.
i
Got it. The
postprocess
directive should work here then
Here's an example that parses the JSON out of every test case:
r
So that allows me to see it but creates 2 other issues: 1. It's only running the first row and then stopping 2. The actual comparison is not working these should not be passing
i
Just made a commit that fixes the display, but lets you use the JSON object directly in your tests: I think this should fix the problem and allow you to remove the postprocessing. Otherwise in order to compare values within the output, you'd have to parse the JSON again 😬
r
Got it. If I didn't use JSON in the test what would it automatically be comparing? Just the first message?
i
are you able to share your test?
just hard to say without knowing more about how you've set up the test
but basically, the
postprocess
directive that we added makes the LLM output a string. So your tests will have to assume the output is a string rather than an object
r
yeah DMing you
i
cool just troubleshooting here right now. One of the first things I notice is that the column name should be
__expected
, with 2 underscores . That's why the tests aren't actually running
made a copy and going to check if my fix above solves the issue
ok, with the
__expected
fix and the changes just released in version
0.26.3
, your setup should properly compare (remove the
postprocess
directive)
more could be done to improve comparison for actual JSON objects, but hopefully this at least unblocks you
r
Word thank you.
Just for my own understanding before it was trying to treat the each row as text? If so why did it auto cast it to a JSON object?
I'm asking because it looks like it's automatically grabbing the object inside of function_call
i
yeah, if
function_call
is present then the provider returns the
function_call
object, not the full response object . Shouldn't be too difficult to modify your test cases accordingly 🙂
The problem was that it was expecting a string but receiving a JSON object. The OpenAI provider normally returns a string, except for in the case of function calls
r
Got it my $0.000002 is that I'd prefer if the default behavior was predictable. That way I can generate a test dataset that contains examples when the content is not null and the functional call is not auto.
i
Makes sense. I can add a flag that causes the OpenAI provider to return the raw output
{content, function_call}
. Would that help? I can't change the default behavior for now because it would break pretty much everyone
2 Views