Hello We have setup a monitoring dashboard on grafana for pi Apache Pinot #troubleshooting

Hello, We have setup a monitoring dashboard on gra...

Tanmay Movva

03/24/2021, 11:48 AM

Hello, We have setup a monitoring dashboard on grafana for pinot. We are currently using the image with

latest

tag and we observe that few metrics related to query response time have been coming as

. We didn’t change any config on pinot or the grafana query, so wanted to know if there was any breaking change merged recently? Here is a screen shot of the dashboard.

Tanmay Movva

03/24/2021, 11:54 AM

Affected Metrics -

Copy code

pinot_broker_queryExecution_*
pinot_server_freshnessLagMs_*
pinot_server_totalQueryTime_*
pinot_server_queryProcessing_*

Kishore G

03/24/2021, 2:04 PM

Can you check the JMX metric names. There was a change in the metric name

Kishore G

03/24/2021, 2:04 PM

@Xiang Fu

Xiang Fu

03/24/2021, 3:07 PM

I will check, could be related to the recent metrics plugin extraction @Jack just fyi

Xiang Fu

03/24/2021, 4:05 PM

@Tanmay Movva do you know which image you are using previously? Or when you restarted your deployment before this restart ?

Tanmay Movva

03/24/2021, 4:15 PM

We are/were using the image with

latest

tag only. From the metrics dashboard, we are not able to see the metrics after 17th March(We restarted on that day).

Xiang Fu

03/24/2021, 4:16 PM

I will check on that

Tanmay Movva

03/24/2021, 4:16 PM

Can you check the JMX metric names. There was a change in the metric name

Checked for alternate metrics for those but didn’t find any.

Xiang Fu

03/24/2021, 4:26 PM

did you see anything when you run

curl localhost:8080/jmx

in the container?

Xiang Fu

03/24/2021, 4:29 PM

also are the controller metrics there?

Tanmay Movva

03/24/2021, 4:32 PM

also are the controller metrics there?

Yes.

Xiang Fu

03/24/2021, 4:32 PM

hmm, controller are also restarted?

Tanmay Movva

03/24/2021, 4:34 PM

They were restarted, but not sure if the image was pulled for them. Will force a image pull for all the services once.

Jack

03/24/2021, 4:46 PM

I don’t think it’s related to metrics plugin extraction. The latest official release was cut based on the commit hash on 02/24 while metric plugin extraction was merged 03/08

Tanmay Movva

03/24/2021, 4:48 PM

hmm, controller are also restarted?

Just did this, don’t think controller metrics are affected. I randomly checked few metrics and they were coming in. Any specific metrics to check for?

Tanmay Movva

03/24/2021, 4:50 PM

he latest official release was cut based on the commit hash on 02/24 while metric plugin extraction was merged 03/08

We are not using the officially released tag yet. We are using the

latest

tag for pinot image. Is

0.7.0

tag released?

Jack

03/24/2021, 4:50 PM

So the table query latency should be emitted from pinot-broker right? I think you should take a look at the health of broker metric

Tanmay Movva

03/24/2021, 4:53 PM

I can see the QPS metrics from broker but not query latency metrics.

Tanmay Movva

03/24/2021, 4:53 PM

And helix_connected_value, Status of broker on UI all are OK. Queries are also being processed.

Xiang Fu

03/24/2021, 4:55 PM

can u list the metrics missing, then I can check

Tanmay Movva

03/24/2021, 4:56 PM

Hmm..Will try to get list of all metrics, but from the panels I setup in the dashboard, these are the metrics missing currently. https://apache-pinot.slack.com/archives/C011C9JHN7R/p1616586877125400?thread_ts=1616586488.125300&cid=C011C9JHN7R

Xiang Fu

03/24/2021, 6:25 PM

hmm, I tried latest image and found that those metrics are there:

Copy code

root@pinot-server-0:/opt/pinot# curl localhost:8080/jmx|grep freshnessLagMs
pinot_server_freshnessLagMs_MeanRate Attribute exposed for management ("org.apache.pinot.common.metrics"<type="ServerMetrics", name="pinot.server.airlineStatsAvro_REALTIME.freshnessLagMs"><>MeanRate)
# TYPE pinot_server_freshnessLagMs_MeanRate untyped
pinot_server_freshnessLagMs_MeanRate{table="airlineStatsAvro",tableType="REALTIME",} 0.0
pinot_server_freshnessLagMs_MeanRate{table="airlineStats",tableType="REALTIME",} 0.0
# HELP pinot_server_freshnessLagMs_OneMinuteRate Attribute exposed for management ("org.apache.pinot.common.metrics"<type="ServerMetrics", name="pinot.server.airlineStatsAvro_REALTIME.freshnessLagMs"><>OneMinuteRate)
# TYPE pinot_server_freshnessLagMs_OneMinuteRate untyped
pinot_server_freshnessLagMs_OneMinuteRate{table="airlineStatsAvro",tableType="REALTIME",} 0.0
pinot_server_freshnessLagMs_OneMinuteRate{table="airlineStats",tableType="REALTIME",} 0.0
# HELP pinot_server_freshnessLagMs_StdDev Attribute exposed for management ("org.apache.pinot.common.metrics"<type="ServerMetrics", name="pinot.server.airlineStatsAvro_REALTIME.freshnessLagMs"><>StdDev)
# TYPE pinot_server_freshnessLagMs_StdDev untyped
pinot_server_freshnessLagMs_StdDev{table="airlineStatsAvro",tableType="REALTIME",} 0.0
pinot_server_freshnessLagMs_StdDev{table="airlineStats",tableType="REALTIME",} 0.0
# HELP pinot_server_freshnessLagMs_95thPercentile Attribute exposed for management ("org.apache.pinot.common.metrics"<type="ServerMetrics", name="pinot.server.airlineStatsAvro_REALTIME.freshnessLagMs"><>95thPercentile)
# TYPE pinot_server_freshnessLagMs_95thPercentile untyped
pinot_server_freshnessLagMs_95thPercentile{table="airlineStatsAvro",tableType="REALTIME",} 0.0
pinot_server_freshnessLagMs_95thPercentile{table="airlineStats",tableType="REALTIME",} 0.0
# HELP pinot_server_freshnessLagMs_99thPercentile Attribute exposed for management ("org.apache.pinot.common.metrics"<type="ServerMetrics", name="pinot.server.airlineStatsAvro_REALTIME.freshnessLagMs"><>99thPercentile)
# TYPE pinot_server_freshnessLagMs_99thPercentile untyped
pinot_server_freshnessLagMs_99thPercentile{table="airlineStatsAvro",tableType="REALTIME",} 0.0
pinot_server_freshnessLagMs_99thPercentile{table="airlineStats",tableType="REALTIME",} 0.0
# HELP pinot_server_freshnessLagMs_Mean Attribute exposed for management ("org.apache.pinot.common.metrics"<type="ServerMetrics", name="pinot.server.airlineStatsAvro_REALTIME.freshnessLagMs"><>Mean)
# TYPE pinot_server_freshnessLagMs_Mean untyped
pinot_server_freshnessLagMs_Mean{table="airlineStatsAvro",tableType="REALTIME",} 0.0
pinot_server_freshnessLagMs_Mean{table="airlineStats",tableType="REALTIME",} 0.0
100  180k  100  180k    0     0   204k  # HELP pinot_server_freshnessLagMs_FifteenMinuteRate Attribute exposed for management ("org.apache.pinot.common.metrics"<type="ServerMetrics", name="pinot.server.airlineStatsAvro_REALTIME.freshnessLagMs"><>FifteenMinuteRate)
    # TYPE pinot_server_freshnessLagMs_FifteenMinuteRate untyped
0 pinot_server_freshnessLagMs_FifteenMinuteRate{table="airlineStatsAvro",tableType="REALTIME",} 0.0
-pinot_server_freshnessLagMs_FifteenMinuteRate{table="airlineStats",tableType="REALTIME",} 0.0
-# HELP pinot_server_freshnessLagMs_Min Attribute exposed for management ("org.apache.pinot.common.metrics"<type="ServerMetrics", name="pinot.server.airlineStatsAvro_REALTIME.freshnessLagMs"><>Min)
:-# TYPE pinot_server_freshnessLagMs_Min untyped
-:pinot_server_freshnessLagMs_Min{table="airlineStatsAvro",tableType="REALTIME",} 0.0
-pinot_server_freshnessLagMs_Min{table="airlineStats",tableType="REALTIME",} 0.0
- -# HELP pinot_server_freshnessLagMs_Count Attribute exposed for management ("org.apache.pinot.common.metrics"<type="ServerMetrics", name="pinot.server.airlineStatsAvro_REALTIME.freshnessLagMs"><>Count)
-:--:-- --:# TYPE pinot_server_freshnessLagMs_Count untyped
--pinot_server_freshnessLagMs_Count{table="airlineStatsAvro",tableType="REALTIME",} 0.0
:pinot_server_freshnessLagMs_Count{table="airlineStats",tableType="REALTIME",} 0.0
-- # HELP pinot_server_freshnessLagMs_999thPercentile Attribute exposed for management ("org.apache.pinot.common.metrics"<type="ServerMetrics", name="pinot.server.airlineStatsAvro_REALTIME.freshnessLagMs"><>999thPercentile)
 2# TYPE pinot_server_freshnessLagMs_999thPercentile untyped
04pinot_server_freshnessLagMs_999thPercentile{table="airlineStatsAvro",tableType="REALTIME",} 0.0
k
pinot_server_freshnessLagMs_999thPercentile{table="airlineStats",tableType="REALTIME",} 0.0
# HELP pinot_server_freshnessLagMs_50thPercentile Attribute exposed for management ("org.apache.pinot.common.metrics"<type="ServerMetrics", name="pinot.server.airlineStatsAvro_REALTIME.freshnessLagMs"><>50thPercentile)
# TYPE pinot_server_freshnessLagMs_50thPercentile untyped
pinot_server_freshnessLagMs_50thPercentile{table="airlineStatsAvro",tableType="REALTIME",} 0.0
pinot_server_freshnessLagMs_50thPercentile{table="airlineStats",tableType="REALTIME",} 0.0
# HELP pinot_server_freshnessLagMs_75thPercentile Attribute exposed for management ("org.apache.pinot.common.metrics"<type="ServerMetrics", name="pinot.server.airlineStatsAvro_REALTIME.freshnessLagMs"><>75thPercentile)
# TYPE pinot_server_freshnessLagMs_75thPercentile untyped
pinot_server_freshnessLagMs_75thPercentile{table="airlineStatsAvro",tableType="REALTIME",} 0.0
pinot_server_freshnessLagMs_75thPercentile{table="airlineStats",tableType="REALTIME",} 0.0
# HELP pinot_server_freshnessLagMs_FiveMinuteRate Attribute exposed for management ("org.apache.pinot.common.metrics"<type="ServerMetrics", name="pinot.server.airlineStatsAvro_REALTIME.freshnessLagMs"><>FiveMinuteRate)
# TYPE pinot_server_freshnessLagMs_FiveMinuteRate untyped
pinot_server_freshnessLagMs_FiveMinuteRate{table="airlineStatsAvro",tableType="REALTIME",} 0.0
pinot_server_freshnessLagMs_FiveMinuteRate{table="airlineStats",tableType="REALTIME",} 0.0
# HELP pinot_server_freshnessLagMs_98thPercentile Attribute exposed for management ("org.apache.pinot.common.metrics"<type="ServerMetrics", name="pinot.server.airlineStatsAvro_REALTIME.freshnessLagMs"><>98thPercentile)
# TYPE pinot_server_freshnessLagMs_98thPercentile untyped
pinot_server_freshnessLagMs_98thPercentile{table="airlineStatsAvro",tableType="REALTIME",} 0.0
pinot_server_freshnessLagMs_98thPercentile{table="airlineStats",tableType="REALTIME",} 0.0
# HELP pinot_server_freshnessLagMs_Max Attribute exposed for management ("org.apache.pinot.common.metrics"<type="ServerMetrics", name="pinot.server.airlineStatsAvro_REALTIME.freshnessLagMs"><>Max)
# TYPE pinot_server_freshnessLagMs_Max untyped
pinot_server_freshnessLagMs_Max{table="airlineStatsAvro",tableType="REALTIME",} 0.0
pinot_server_freshnessLagMs_Max{table="airlineStats",tableType="REALTIME",} 0.0

Xiang Fu

03/24/2021, 6:26 PM

Copy code

root@pinot-server-0:/opt/pinot# curl localhost:8080/jmx|grep totalQueryTime
pinot_server_totalQueryTime_Count Attribute exposed for management ("org.apache.pinot.common.metrics"<type="ServerMetrics", name="pinot.server.simpleWebsite_OFFLINE.totalQueryTime"><>Count)
# TYPE pinot_server_totalQueryTime_Count untyped
pinot_server_totalQueryTime_Count{table="simpleWebsite",tableType="OFFLINE",} 0.0
pinot_server_totalQueryTime_Count{table="covid19_confirmed_us",tableType="OFFLINE",} 0.0
pinot_server_totalQueryTime_Count{table="covid19_deaths_us",tableType="OFFLINE",} 0.0
pinot_server_totalQueryTime_Count{table="airlineStatsAvro",tableType="REALTIME",} 0.0
pinot_server_totalQueryTime_Count{table="covid19_confirmed",tableType="us",} 0.0
pinot_server_totalQueryTime_Count{table="airlineStats",tableType="REALTIME",} 0.0
pinot_server_totalQueryTime_Count{table="complexWebsite",tableType="OFFLINE",} 0.0
# HELP pinot_server_totalQueryTime_FifteenMinuteRate Attribute exposed for management ("org.apache.pinot.common.metrics"<type="ServerMetrics", name="pinot.server.simpleWebsite_OFFLINE.totalQueryTime"><>FifteenMinuteRate)
# TYPE pinot_server_totalQueryTime_FifteenMinuteRate untyped
pinot_server_totalQueryTime_FifteenMinuteRate{table="simpleWebsite",tableType="OFFLINE",} 0.0
pinot_server_totalQueryTime_FifteenMinuteRate{table="covid19_confirmed_us",tableType="OFFLINE",} 0.0
pinot_server_totalQueryTime_FifteenMinuteRate{table="covid19_deaths_us",tableType="OFFLINE",} 0.0
pinot_server_totalQueryTime_FifteenMinuteRate{table="airlineStatsAvro",tableType="REALTIME",} 0.0
pinot_server_totalQueryTime_FifteenMinuteRate{table="covid19_confirmed",tableType="us",} 0.0
pinot_server_totalQueryTime_FifteenMinuteRate{table="airlineStats",tableType="REALTIME",} 0.0
pinot_server_totalQueryTime_FifteenMinuteRate{table="complexWebsite",tableType="OFFLINE",} 0.0
# HELP pinot_server_totalQueryTime_Max Attribute exposed for management ("org.apache.pinot.common.metrics"<type="ServerMetrics", name="pinot.server.simpleWebsite_OFFLINE.totalQueryTime"><>Max)
# TYPE pinot_server_totalQueryTime_Max untyped
pinot_server_totalQueryTime_Max{table="simpleWebsite",tableType="OFFLINE",} 0.0
pinot_server_totalQueryTime_Max{table="covid19_confirmed_us",tableType="OFFLINE",} 0.0
pinot_server_totalQueryTime_Max{table="covid19_deaths_us",tableType="OFFLINE",} 0.0
pinot_server_totalQueryTime_Max{table="airlineStatsAvro",tableType="REALTIME",} 0.0
pinot_server_totalQueryTime_Max{table="covid19_confirmed",tableType="us",} 0.0
pinot_server_totalQueryTime_Max{table="airlineStats",tableType="REALTIME",} 0.0
pinot_server_totalQueryTime_Max{table="complexWebsite",tableType="OFFLINE",} 0.0
# HELP pinot_server_totalQueryTime_50thPercentile Attribute exposed for management ("org.apache.pinot.common.metrics"<type="ServerMetrics", name="pinot.server.simpleWebsite_OFFLINE.totalQueryTime"><>50thPercentile)
# TYPE pinot_server_totalQueryTime_50thPercentile untyped
pinot_server_totalQueryTime_50thPercentile{table="simpleWebsite",tableType="OFFLINE",} 0.0
pinot_server_totalQueryTime_50thPercentile{table="covid19_confirmed_us",tableType="OFFLINE",} 0.0
pinot_server_totalQueryTime_50thPercentile{table="covid19_deaths_us",tableType="OFFLINE",} 0.0
pinot_server_totalQueryTime_50thPercentile{table="airlineStatsAvro",tableType="REALTIME",} 0.0
pinot_server_totalQueryTime_50thPercentile{table="covid19_confirmed",tableType="us",} 0.0
pinot_server_totalQueryTime_50thPercentile{table="airlineStats",tableType="REALTIME",} 0.0
pinot_server_totalQueryTime_50thPercentile{table="complexWebsite",tableType="OFFLINE",} 0.0
# HELP pinot_server_totalQueryTime_98thPercentile Attribute exposed for management ("org.apache.pinot.common.metrics"<type="ServerMetrics", name="pinot.server.simpleWebsite_OFFLINE.totalQueryTime"><>98thPercentile)
# TYPE pinot_server_totalQueryTime_98thPercentile untyped
pinot_server_totalQueryTime_98thPercentile{table="simpleWebsite",tableType="OFFLINE",} 0.0
pinot_server_totalQueryTime_98thPercentile{table="covid19_confirmed_us",tableType="OFFLINE",} 0.0
pinot_server_totalQueryTime_98thPercentile{table="covid19_deaths_us",tableType="OFFLINE",} 0.0
pinot_server_totalQueryTime_98thPercentile{table="airlineStatsAvro",tableType="REALTIME",} 0.0
pinot_server_totalQueryTime_98thPercentile{table="covid19_confirmed",tableType="us",} 0.0
pinot_server_totalQueryTime_98thPercentile{table="airlineStats",tableType="REALTIME",} 0.0
pinot_server_totalQueryTime_98thPercentile{table="complexWebsite",tableType="OFFLINE",} 0.0
# HELP pinot_server_totalQueryTime_75thPercentile Attribute exposed for management ("org.apache.pinot.common.metrics"<type="ServerMetrics", name="pinot.server.simpleWebsite_OFFLINE.totalQueryTime"><>75thPercentile)
# TYPE pinot_server_totalQueryTime_75thPercentile untyped
pinot_server_totalQueryTime_75thPercentile{table="simpleWebsite",tableType="OFFLINE",} 0.0
pinot_server_totalQueryTime_75thPercentile{table="covid19_confirmed_us",tableType="OFFLINE",} 0.0
pinot_server_totalQueryTime_75thPercentile{table="covid19_deaths_us",tableType="OFFLINE",} 0.0
pinot_server_totalQueryTime_75thPercentile{table="airlineStatsAvro",tableType="REALTIME",} 0.0
pinot_server_totalQueryTime_75thPercentile{table="covid19_confirmed",tableType="us",} 0.0
pinot_server_totalQueryTime_75thPercentile{table="airlineStats",tableType="REALTIME",} 0.0
pinot_server_totalQueryTime_75thPercentile{table="complexWebsite",tableType="OFFLINE",} 0.0
# HELP pinot_server_totalQueryTime_FiveMinuteRate Attribute exposed for management ("org.apache.pinot.common.metrics"<type="ServerMetrics", name="pinot.server.simpleWebsite_OFFLINE.totalQueryTime"><>FiveMinuteRate)
# TYPE pinot_server_totalQueryTime_FiveMinuteRate untyped
pinot_server_totalQueryTime_FiveMinuteRate{table="simpleWebsite",tableType="OFFLINE",} 0.0
pinot_server_totalQueryTime_FiveMinuteRate{table="covid19_confirmed_us",tableType="OFFLINE",} 0.0
pinot_server_totalQueryTime_FiveMinuteRate{table="covid19_deaths_us",tableType="OFFLINE",} 0.0
pinot_server_totalQueryTime_FiveMinuteRate{table="airlineStatsAvro",tableType="REALTIME",} 0.0
pinot_server_totalQueryTime_FiveMinuteRate{table="covid19_confirmed",tableType="us",} 0.0
pinot_server_totalQueryTime_FiveMinuteRate{table="airlineStats",tableType="REALTIME",} 0.0
pinot_server_totalQueryTime_FiveMinuteRate{table="complexWebsite",tableType="OFFLINE",} 0.0
# HELP pinot_server_totalQueryTime_StdDev Attribute exposed for management ("org.apache.pinot.common.metrics"<type="ServerMetrics", name="pinot.server.simpleWebsite_OFFLINE.totalQueryTime"><>StdDev)
# TYPE pinot_server_totalQueryTime_StdDev untyped
pinot_server_totalQueryTime_StdDev{table="simpleWebsite",tableType="OFFLINE",} 0.0
pinot_server_totalQueryTime_StdDev{table="covid19_confirmed_us",tableType="OFFLINE",} 0.0
pinot_server_totalQueryTime_StdDev{table="covid19_deaths_us",tableType="OFFLINE",} 0.0
pinot_server_totalQueryTime_StdDev{table="airlineStatsAvro",tableType="REALTIME",} 0.0
pinot_server_totalQueryTime_StdDev{table="covid19_confirmed",tableType="us",} 0.0
pinot_server_totalQueryTime_StdDev{table="airlineStats",tableType="REALTIME",} 0.0
pinot_server_totalQueryTime_StdDev{table="complexWebsite",tableType="OFFLINE",} 0.0
100  180k  100  180k    0     0   204k      0 --:--:-- --:--:-- --:--:--  204k
# HELP pinot_server_totalQueryTime_OneMinuteRate Attribute exposed for management ("org.apache.pinot.common.metrics"<type="ServerMetrics", name="pinot.server.simpleWebsite_OFFLINE.totalQueryTime"><>OneMinuteRate)
# TYPE pinot_server_totalQueryTime_OneMinuteRate untyped
pinot_server_totalQueryTime_OneMinuteRate{table="simpleWebsite",tableType="OFFLINE",} 0.0
pinot_server_totalQueryTime_OneMinuteRate{table="covid19_confirmed_us",tableType="OFFLINE",} 0.0
pinot_server_totalQueryTime_OneMinuteRate{table="covid19_deaths_us",tableType="OFFLINE",} 0.0
pinot_server_totalQueryTime_OneMinuteRate{table="airlineStatsAvro",tableType="REALTIME",} 0.0
pinot_server_totalQueryTime_OneMinuteRate{table="covid19_confirmed",tableType="us",} 0.0
pinot_server_totalQueryTime_OneMinuteRate{table="airlineStats",tableType="REALTIME",} 0.0
pinot_server_totalQueryTime_OneMinuteRate{table="complexWebsite",tableType="OFFLINE",} 0.0
# HELP pinot_server_totalQueryTime_Min Attribute exposed for management ("org.apache.pinot.common.metrics"<type="ServerMetrics", name="pinot.server.simpleWebsite_OFFLINE.totalQueryTime"><>Min)
# TYPE pinot_server_totalQueryTime_Min untyped
pinot_server_totalQueryTime_Min{table="simpleWebsite",tableType="OFFLINE",} 0.0
pinot_server_totalQueryTime_Min{table="covid19_confirmed_us",tableType="OFFLINE",} 0.0
pinot_server_totalQueryTime_Min{table="covid19_deaths_us",tableType="OFFLINE",} 0.0
pinot_server_totalQueryTime_Min{table="airlineStatsAvro",tableType="REALTIME",} 0.0
pinot_server_totalQueryTime_Min{table="covid19_confirmed",tableType="us",} 0.0
pinot_server_totalQueryTime_Min{table="airlineStats",tableType="REALTIME",} 0.0
pinot_server_totalQueryTime_Min{table="complexWebsite",tableType="OFFLINE",} 0.0
# HELP pinot_server_totalQueryTime_Mean Attribute exposed for management ("org.apache.pinot.common.metrics"<type="ServerMetrics", name="pinot.server.simpleWebsite_OFFLINE.totalQueryTime"><>Mean)
# TYPE pinot_server_totalQueryTime_Mean untyped
pinot_server_totalQueryTime_Mean{table="simpleWebsite",tableType="OFFLINE",} 0.0
pinot_server_totalQueryTime_Mean{table="covid19_confirmed_us",tableType="OFFLINE",} 0.0
pinot_server_totalQueryTime_Mean{table="covid19_deaths_us",tableType="OFFLINE",} 0.0
pinot_server_totalQueryTime_Mean{table="airlineStatsAvro",tableType="REALTIME",} 0.0
pinot_server_totalQueryTime_Mean{table="covid19_confirmed",tableType="us",} 0.0
pinot_server_totalQueryTime_Mean{table="airlineStats",tableType="REALTIME",} 0.0
pinot_server_totalQueryTime_Mean{table="complexWebsite",tableType="OFFLINE",} 0.0
# HELP pinot_server_totalQueryTime_MeanRate Attribute exposed for management ("org.apache.pinot.common.metrics"<type="ServerMetrics", name="pinot.server.simpleWebsite_OFFLINE.totalQueryTime"><>MeanRate)
# TYPE pinot_server_totalQueryTime_MeanRate untyped
pinot_server_totalQueryTime_MeanRate{table="simpleWebsite",tableType="OFFLINE",} 0.0
pinot_server_totalQueryTime_MeanRate{table="covid19_confirmed_us",tableType="OFFLINE",} 0.0
pinot_server_totalQueryTime_MeanRate{table="covid19_deaths_us",tableType="OFFLINE",} 0.0
pinot_server_totalQueryTime_MeanRate{table="airlineStatsAvro",tableType="REALTIME",} 0.0
pinot_server_totalQueryTime_MeanRate{table="covid19_confirmed",tableType="us",} 0.0
pinot_server_totalQueryTime_MeanRate{table="airlineStats",tableType="REALTIME",} 0.0
pinot_server_totalQueryTime_MeanRate{table="complexWebsite",tableType="OFFLINE",} 0.0
# HELP pinot_server_totalQueryTime_95thPercentile Attribute exposed for management ("org.apache.pinot.common.metrics"<type="ServerMetrics", name="pinot.server.simpleWebsite_OFFLINE.totalQueryTime"><>95thPercentile)
# TYPE pinot_server_totalQueryTime_95thPercentile untyped
pinot_server_totalQueryTime_95thPercentile{table="simpleWebsite",tableType="OFFLINE",} 0.0
pinot_server_totalQueryTime_95thPercentile{table="covid19_confirmed_us",tableType="OFFLINE",} 0.0
pinot_server_totalQueryTime_95thPercentile{table="covid19_deaths_us",tableType="OFFLINE",} 0.0
pinot_server_totalQueryTime_95thPercentile{table="airlineStatsAvro",tableType="REALTIME",} 0.0
pinot_server_totalQueryTime_95thPercentile{table="covid19_confirmed",tableType="us",} 0.0
pinot_server_totalQueryTime_95thPercentile{table="airlineStats",tableType="REALTIME",} 0.0
pinot_server_totalQueryTime_95thPercentile{table="complexWebsite",tableType="OFFLINE",} 0.0

Xiang Fu

03/24/2021, 6:26 PM

can you check if the jmx server are reporting those metrics?

Xiang Fu

03/24/2021, 6:26 PM

also check if your prometheus metrics exporter are deployed

Tanmay Movva

03/24/2021, 7:46 PM

Those metrics are coming in. But they are coming in as

. They aren’t being updated. Even the metrics you shared have zero value. But when queries are running then execution Time shouldn’t be zero no?

Tanmay Movva

03/24/2021, 7:46 PM

My problem was not that they were missing, but the value of those metrics is always zero. Which wasn’t the case earlier.

Xiang Fu

03/24/2021, 8:33 PM

i see, i will double check

Xiang Fu

03/31/2021, 6:25 PM

https://github.com/apache/incubator-pinot/pull/6734

Tanmay Movva

04/01/2021, 12:13 PM

@Xiang Fu is the latest image published after this merge? If yes, will rotate the pods in our setup. Also hoping this will be part of 0.7.0 release.

Xiang Fu

04/01/2021, 5:09 PM

not yet

Xiang Fu

04/01/2021, 5:09 PM

the image is built daily

Xiang Fu

04/01/2021, 5:10 PM

You can take it maybe after 2 hours

Open in Slack

Previous Next