I'd like to run some evals with GPT-5-pro, and I'd say usually 50% of them error out. I get a variety of errors: API call error: Error: Request failed after 4 retries: TypeError: fetch failed (Cause: Error: getaddrinfo ENOTFOUND api.openai.com) API call error: Error: Request failed after 4 retries: Error: Request timed out after 300000 ms API call error: Error: Error parsing response from https://api.openai.com/v1/responses: Unexpected token '<', " <h"... is not valid JSON. Received text: 502 Bad Gateway 502 Bad Gateway cloudflare API call error: Error: Error parsing response from https://api.openai.com/v1/responses: Unexpected token 'u', "upstream c"... is not valid JSON. Received text: upstream connect error or disconnect/reset before headers. reset reason: connection termination Presumably I can resolve one of these by increasing the amount of time per completion, but the other ones... Curious if there are tips for working with models like these. Thanks!

I think your best bet may be to just retry errored test cases, since this seems to be due to the target itself (https://www.promptfoo.dev/docs/usage/command-line/#retry-errors)

We merged and deployed https://github.com/promptfoo/promptfoo/pull/6147 which should help!