This message was deleted.

Actually due to this there is no stability to our druid monitoring peace

we are using prometheus exporter + aggregating metrics e.g. all labels of task_id are removed, also interval, so we compute some statistics over those(min/max/percentiles etc) Otherwise number of metrics is huge and increasing

@Igor Berman can we have a call if possible to you need to understand how you implemented... it will be very helpful

At basic level: prometheus exporter for all services except for peons for peons: prometheus exporter with push gateway to some ‘staging prometheus cluster’, which has recording rules to aggregate metrics from peons. Raw peon metrics has very low retention + cleanup. Then those metrics are scraped by production cluster with higher retention rules

@Igor Berman Any document is there can you atleast please point me to that