This message was deleted.

One idea we have is to maintain a central list of known flaky tests, that can be accessed and updated via API. All builds would check the list of known flaky tests and ignore those. That way a flaky test could be added to the list and instantly all builds would know about it and avoid it.

There is a gradle plugin to retry failed tests, which might be worth a look. We're using it successfully for integration tests. I wouldn't use it without having some other means of identifying and fixing the problematic tests, though, but you already have that it seems.

Hi! At my company, we're using Test Retry with GE to identify flaky tests, but still let them fail the build on CI. Then developers can see PR checks failed because of a flaky test. A con is that the flakiest ones still fail a lot of builds, but at least get noticed and fixed faster (usually in proportion to how flaky they are and the chaos they might be causing 🙂). But I've been thinking about trying to catch flaky tests before they reach the main branch. Maybe running new and modified tests multiple times as part of PR checks? Wouldn't catch all types of flaky, but still. If anyone went down that road, I'd love to hear about it

@Brian Stewart I think a way to auto-ignore tests that have recently been flaky and are still under investigation would be most helpful. I can’t promise anything yet, as far as Gradle Enterprise is concerned, but it’s definitely something we’d like to explore building as part of the product. In the meantime, assuming there’s an API to lookup a flaky test ticket for a given test, you could build a JUnit Jupiter extension based on

TestExecutionExceptionHandler

that would swallow the exception based on that condition.

@Gabriel Feo one thing that companies do is to run all new tests a specific number of times (like 100) before integrating on the main branch to ensure they are not flaky.

@no, do you know if there's a specific tool or plugin for that? I presume I could set it up with junit somehow. There's RepeatedTest but would have to annotate each method with it before running

Yeah that's the only one I'm aware of. Something like the

ShampooRule

also exists: https://gist.github.com/JakeWharton/7fe7deb1f7f4a795c120

Thanks for the feedback everyone! Much appreciated, this gives us something to go on. 🙏