When a provider throws rate limit error or token limit error,it will be handled by promotfoo internally and assertion process will be skipped for that.How we can catch that and apply assertion logic to that? Can anyone help regarding this. Thank you!

Hi @Nithya, that's a great question. When a provider returns an error (rate limit, token limit), promptfoo marks the test case as failed and skips all assertions. A couple things to note: - Rate limits are already auto-retried with exponential backoff (up to ~7 attempts). You only see an error if all retries fail. You can tune --max-concurrency and --delay to reduce rate limiting, see our rate limits page in the docs https://www.promptfoo.dev/docs/configuration/rate-limits/. - Token limit errors aren't retried since they're deterministic. To apply assertion logic to these errors, the main approach is to intercept the error before it reaches the evaluator, converting it into output so assertions can run on it. How you do this depends on your setup: 1. If you're using the HTTP provider, transformResponse can catch non-429 error responses (400, 500, etc.) and convert them to output 2. If you're using a built-in provider (like openai:chat:gpt-4o), a small wrapper provider using loadApiProvider can catch all error types What provider/setup are you using?

Thanks a lot I am using HTTP provider

Since you're using the HTTP provider, you can handle this with transformResponse in your config. It runs before the evaluator sees the response, so you can convert error HTTP responses into output that your assertions can run on:

providers:
    - id: https://your-api.com/endpoint
      config:
        # ... your existing config ...
        transformResponse: |
          (json, text, context) => {
            const status = context?.response?.status;
            if (status >= 400) {
              return { output: `PROVIDER_ERROR (HTTP ${status}): ${json?.error || text}` };
            }
            return { output: json?.output || text };
          }

The HTTP provider accepts all status codes by default, so transformResponse will run on 400s, 500s, etc. Then you can write assertions against the error output like normal. One exception: HTTP 429 (rate limit) responses are intercepted by the retry layer before reaching transformResponse, so this won't catch those, but rate limits are auto-retried anyway so I'm guessing they aren't your main concern. Here are some docs links if you want to dive deeper: - https://www.promptfoo.dev/docs/providers/http/#response-transform - https://www.promptfoo.dev/docs/red-team/troubleshooting/multiple-response-types/

Thanks a lot, But in case of 429 error(rate limit error or resource exhausted case) I want to get count of test cases which got skipped due to that error What can I do in this?

@Nithya Promptfoo already tracks this! When you run an eval, the CLI summary shows errors separately:

Results: 10 passed, 3 failed, ✗ 2 errors (78%)

Errors (like rate limits) are counted distinctly from assertion failures. A few more things that might help: - Re-run only errored tests:` promptfoo eval --retry-errors` will automatically re-run just the test cases that hit provider errors - Filter in the web UI:

promptfoo view

has a display mode dropdown where you can filter to "Errors" only - Export and count:

promptfoo eval -o results.json

, then filter by

failureReason === 2

(that's the error code, distinct from 1 which is assertion failure)

Yeah promotfoo tracks,but through my assertion I want to create a file for this such that each and every must be recorded

Assertions won't run on errored responses, but you don't need assertions for this: just export your results and filter them: promptfoo eval -o results.json Every test case is recorded in that file, including ones that errored. Error results have failureReason: 2 and include the error message.

Okay thank you, so much I will try this approach to filter it out.

Absolutely, always happy to help