Does anybody have tips or tricks when it comes to diagnosing issues of builds failing on ci? We'd like to start tracking issues where daemons dissapear, or are shutdown due to crashing, or crash due to 'user error' (tests failing, compile failing) and other causes of ci flakiness. Are there apis or files we can inspect to validate run results after the fact? Or do we just have to search thru build logs and look for key words?

If you have daemons disappearing it is usually down to memory issues. One way to workaround is to use forkEvery with a low setting when in CI. You can use the org.ysb33r.cloudci plugin to set it conditionally. Also having build scans can also help in diagnostics.