Is it possible to exclude errors from Red Teaming Vulnerability Reports? Right now errors are counted as "Failures" on vulnerability reports, which tend to skew the results a bit, as they're not always representative of actual failures, but possibly just timeouts or other issues. I currently have to go in and manually mark each error as a "success" to make the vulnerability reports look more correct, which also isn't really accurate. It'd be great to just be able to fully exclude errors if possible

Heyo! Sorry for the delay. Have you looked into response transforms to map errors from the target application (e.g. error 500) as a regular rejection through the 'output' field? This is pretty much the workaround for the OSS version.

The next version will exclude errors from reporting 🙂 https://github.com/promptfoo/promptfoo/pull/5271

@ian fantastic! Thank you! And @Tabs [Promptfoo] that's a good note, I'll look more into that, I've used assertions to try to catch that but not response transforms, I'll look into that if I run into this error outside the fix that Ian mentioned. Thanks!

No worries! Thanks to @shuo0446 for giving me the info