https://promptfoo.dev/ logo
Issues testing replies that are typed schemas like...
# ❓|questions
s
Hi- can promptfoo do this? I'm not sure what the right abstractions are (maybe providers?) but I want to be able to change some things like the text in the schema descriptions and the text around the template inputs and test if it broke some prior cases that worked. We have JS functions that take business logic inputs and use a schema to get out a typed reply. I didn't see any examples using Instructor in the examples directory. If someone can point me in the right direction on how to use this together I'll add one to the repo.
Copy code
typescript

const client = Instructor({
   client: oai,
    mode: "TOOLS",
  })
const ExampleSchema = z.object({
    size: z.string().describe("size"),
    weight: z.string().describe("weight"),
  })

  return await client.chat.completions.create({
    messages: [
      {
        role: "user",
        content: `${inputString}`,
      },
    ],
    model: "gpt-4-turbo-2024-04-09",
    response_model: {
      schema: ExampleSchema,
      name: "Physical_Info",
    },
  })
}
I'm hoping to use this as a provider basically and pass different input strings to it that are known cases and assert that certain things are true. Yes you can parse out specific JSON keys in asserts, thats NOT the issue- the issue is should I be doing one provider per yaml file since I'm not looking to run all tests against all providers. Is there a way I can use this function to run the tests but also keep the nice html output? Thanks!
I found this page in the docs, this feels like it might be the best fit- getting to leverage promptfoo's awesome assertions while being able to define my own test structure due to having to run those functions to get the outputs https://www.promptfoo.dev/docs/integrations/jest/