Apache Pinot

hi. wondering about composite health endpoint for servicemanager

```  @GET
  @Produces(MediaType.APPLICATION_JSON)
  @Path("/instances")
  @ApiOperation(value = "Get Pinot Instances Status")
  @ApiResponses(value = {@ApiResponse(code = 200, message = "Instance Status"), @ApiResponse(code = 500, message = "Internal server error")})
  public Map&lt;String, PinotInstanceStatus&gt; getPinotAllInstancesStatus() {
    Map&lt;String, PinotInstanceStatus&gt; results = new HashMap&lt;&gt;();
    for (String instanceId : _pinotServiceManager.getRunningInstanceIds()) {
      results.put(instanceId, _pinotServiceManager.getInstanceStatus(instanceId));
    }
    return results;
  }```

seems ^^ could be slightly changed or a similar copy for /health that returns HTTP 200 when all are healthy and something else when not

similar to calling each of <https://github.com/apache/incubator-pinot/pull/5846/files>

agree, we can have a separated `/health` for serviceManager

personally I think SM health should return health of its subordinates in aggregate

as I don't know why it is important to return success if as a process it is down. Right now, we rely a lot on knowing things are all on different ports, but if you think of the unit of SM, it represents a composite service correct?

for example, in zipkin we have a composite health endpoint because if its connection to storage is busted it is not servicable as a process

we sometimes start a second listener depending on config, but because we use armeria, generally we don't need a new port for every feature running

and when you take network ports out of it, it simplifies health. is the unit servicable or not basically. if it is composite you can see what things actually work, if nice json is used. spring boot does this by default but we wrote our own for perf.

<https://github.com/openzipkin/zipkin/blob/master/zipkin-server/src/main/java/zipkin2/server/internal/health/ComponentHealth.java>

<https://github.com/openzipkin/zipkin/blob/master/zipkin-server/src/main/java/zipkin2/server/internal/health/ZipkinHealthController.java>

ack it is a little more complex in impl than we probably need for the pinot side. main thing is to represent the composite of its health so you can know if the process should be in service or not

for example, i noticed one part of process fail in docker due to zip extraction maybe take too long no idea. still passes health check! that's bad as it fails other things. incidentally that fail went away when I flatten classpath, but that's a separate topic

I agree with Adrian, if anything within the subsystem is not healthy, a single endpoint should return the aggregate of all health checks for the process and 5xx when any of those is not healthy.