https://promptfoo.dev/ logo
GPT4 response_format with Pydantic
# ❓|questions
c
I'm mainly interested in using the framework to validate the outputs of OpenAI GPT4 calls with structured outputs. Will there be any documentation around using the framework with Pydantic objects? I saw we can include a config key like here https://www.promptfoo.dev/docs/providers/openai/#using-response_format but it only mentions json_schema. In the Python openai client, we can call the chat completions api using the response format as follows: response = self.client.beta.chat.completions.parse( model=self.model, messages=messages, temperature=0, response_format=PydanticModel, ) Thank you
i
I think the best way to do this would be as a python provider that calls openai with the model you want. https://www.promptfoo.dev/docs/providers/python/ You'd just have to implement
Copy code
py
def call_api(prompt: str) -> ProviderResponse:
and throw an error if the response format is not accurate. Alternatively, you can set
response_format
on the provider config as a json schema. Example of that [here](https://github.com/promptfoo/promptfoo/blob/700904ffa602b687cdf6e6c616e061315e6a5310/examples/openai-structured-output/promptfooconfig.yaml#L9)
c
Thanks Ian, this should be enough to get started! 🙏
There is no planned support for Pydantic objects by any chance?