This message was deleted.

it does seem useful to me. Tagging @Suneet Saldanha who has been doing some work in this area if he has some more thoughts.

@Suneet Saldanha Gentle bump! What do you think about this potential improvement?

Hi @Yuanli Han, have you considered using the Prometheus emitter extension? It exposes metrics on each service in the Prometheus metrics format so that it can be scraped by metric systems

@Suneet Saldanha Yes, the Prometheus emitter extension does have an

exporter

option to expose a specific endpoint for Prometheus to collect metrics from. But I think it's not what I expect to have about the stats API because its metrics format is specific for Prometheus. I expect the stats API to be a general endpoint to retrieve statistics about the cluster, a comprehensive json not detail rows of metrics. It can be similar to cluster stats API of ElasticSearch https://www.elastic.co/guide/en/elasticsearch/reference/7.9/cluster-stats.html

Could you elaborate on why a single endpoint to fetch cluster stats for the whole cluster is needed?

That's a great question. I can see a lot benefits to have a single API to retrieve cluster wide metrics. 1. It'll be convenient to inspect druid cluster. Currently, it cost much effort to observe druid cluster. We have to let each service emit raw metrics to somewhere to collect and store them and then query them. Although all these metrics are from Druid itself. 2. It'll be convenient to integrate with pull or scrape based metrics collection ecosystems. 3. Some overall statistics are already available from router console page but they're retrieved from various queries. I think the cluster observability related pages of Router can also benefit from the stats API cc @Vadim

Are you thinking of exposing new stats or consolidating the stats that the console home page gathers from multiple queries into a single API call or both?

I think we can do both.

Does it make sense to add a cluster

/stats

to allow to retrieve statistics from a cluster wide perspective?

I think the key thing to outline in your feature proposal here is how are stats different from metrics. How should a Druid developer decide whether or not a metric should be made available via the stats api or the metrics endpoint or both.

We can add the API to the Router service

I haven't thought about this deeply enough to know if the router is the best place for this API. Usually the router is pretty light weight and just forwards requests to the right places. If it needs to merge responses from multiple services, then it's footprint could grow.

1. Integration with the pull based metrics receciver to extend the current metrics collection solution which is based on emitter pattern

From my reading most pull based metrics solutions use a pattern to scrape each service. And since Druid has multiple services it makes sense to scrape each service independently instead of relying on the router to coalesce all these metrics from the services and tag the metrics appropriately.

I'm willing to submit a feature proposal if it makes sense.

A feature proposal in github sounds great so others can comment on it! Directionally speaking, I think it may make sense to have an endpoint that summarizes some basic cluster statistics - like the details displayed in the web console. But i don;t think that sort of information can be used for more detailed troubleshooting. Which is why outlining the difference between a cluster "stat" and a "metric" will be important.

But i don;t think that sort of information can be used for more detailed troubleshooting. Which is why outlining the difference between a cluster "stat" and a "metric" will be important.

Yes, I agree it might not help much into detailed troubleshooting. The new endpoint is more likely to include general stats about the cluster.