Hi, trying to scrape metrics on prometheus, gettin...
# troubleshootings
Sumit Singh
08/08/2023, 6:31 AMHi, trying to scrape metrics on prometheus, getting error
expected a valid start token, got "[" ("INVALID") while parsing: "["<http://localhost:8081/jobmanager/metrics> Copy code
[{"id":"Status.JVM.GarbageCollector.PS_MarkSweep.Time"},{"id":"Status.JVM.Memory.Mapped.TotalCapacity"},{"id":"taskSlotsAvailable"},{"id":"taskSlotsTotal"},{"id":"Status.JVM.Memory.Mapped.MemoryUsed"},{"id":"Status.JVM.CPU.Time"},{"id":"Status.JVM.Threads.Count"},{"id":"Status.JVM.Memory.Heap.Committed"},{"id":"Status.JVM.Memory.Metaspace.Committed"},{"id":"Status.JVM.GarbageCollector.PS_MarkSweep.Count"},{"id":"Status.JVM.GarbageCollector.PS_Scavenge.Time"},{"id":"Status.JVM.Memory.Direct.Count"},{"id":"Status.JVM.GarbageCollector.PS_Scavenge.Count"},{"id":"Status.JVM.Memory.NonHeap.Max"},{"id":"numRegisteredTaskManagers"},{"id":"Status.JVM.Memory.NonHeap.Committed"},{"id":"Status.JVM.Memory.NonHeap.Used"},{"id":"Status.JVM.Memory.Metaspace.Max"},{"id":"Status.JVM.Memory.Direct.MemoryUsed"},{"id":"Status.JVM.Memory.Direct.TotalCapacity"},{"id":"numRunningJobs"},{"id":"Status.JVM.ClassLoader.ClassesLoaded"},{"id":"Status.JVM.Memory.Mapped.Count"},{"id":"Status.JVM.Memory.Metaspace.Used"},{"id":"Status.JVM.CPU.Load"},{"id":"Status.JVM.Memory.Heap.Max"},{"id":"Status.JVM.Memory.Heap.Used"},{"id":"Status.JVM.ClassLoader.ClassesUnloaded"}] Copy code
metrics.reporters: prom
metrics.reporter.prom.class: org.apache.flink.metrics.prometheus.PrometheusReporterk
Krutik Gadhiya
08/08/2023, 9:46 AMYou also need to add factory class as bellow
Copy code
metrics.reporters: prom
metrics.reporter.prom.class: org.apache.flink.metrics.prometheus.PrometheusReporter
metrics.reporter.prom.factory.class: org.apache.flink.metrics.prometheus.PrometheusReporterFactorys
Sumit Singh
08/08/2023, 9:51 AMthanks @Krutik Gadhiya, as suggested added factory class as well, but still getting the same error. Below is default prometheus.yml, whats wrong in this any idea ?
Copy code
# my global config
global:
scrape_interval: 15s
evaluation_interval: 15s
# scrape_timeout is set to the global default (10s).
# Alertmanager configuration
alerting:
alertmanagers:
- static_configs:
- targets:
# - alertmanager:9093
# Load rules once and periodically evaluate them according to the global 'evaluation_interval'.
rule_files:
# - "first_rules.yml"
# - "second_rules.yml"
# A scrape configuration containing exactly one endpoint to scrape:
# Here it's Prometheus itself.
# scrape_configs:
# The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.
# - job_name: "prometheus"
# metrics_path defaults to '/metrics'
# scheme defaults to 'http'.
scrape_configs:
- job_name: "flink"
metrics_path: /jobmanager/metrics
static_configs:
- targets:
- "localhost:8081"
- "taskmanager1:8081"
- "taskmanager2:8081"k
Krutik Gadhiya
08/08/2023, 9:53 AMtry without
metrics_paths
Sumit Singh
08/08/2023, 9:58 AMSumit Singh
08/08/2023, 10:08 AMI guess, metrics_path needs to be defined
k
Krutik Gadhiya
08/08/2023, 10:09 AMare propetheus and flink running in same environment?
s
Sumit Singh
08/08/2023, 10:10 AMyes, for now i am testing on my mac
Sumit Singh
08/08/2023, 10:10 AMon Prod, both are running in different EC2, but its configured to scrape data from all EC2s
k
Krutik Gadhiya
08/08/2023, 10:18 AMmetrics_path has default value as /metrics
Krutik Gadhiya
08/08/2023, 10:20 AMalso you need to add
metrics.reporter.prom.port: ports
Sumit Singh
08/08/2023, 10:20 AMDefault port is 8081
k
Krutik Gadhiya
08/08/2023, 10:25 AMi think you need to give a port or port range
i am using something like this, so on the first port 9250 jobmanager's metrics would be exposed and after then each taskmanager's metrics would be available on the ports from 9251
metrics.reporter.prom.port: 9250-9260s
Sumit Singh
08/08/2023, 10:29 AMCan you share prometheus.yml’s flink part as well
k
Krutik Gadhiya
08/08/2023, 10:30 AM Copy code
- job_name: "flink_JobManager"
# metrics_path defaults to '/metrics'
# scheme defaults to 'http'.
static_configs:
- targets: ["localhost:9250"]
- job_name: 'flink_TaskManager'
static_configs:
- targets: ["localhost:9251"]s
Sumit Singh
08/08/2023, 10:38 AMGreat Bro !! Working now Thanks @Krutik Gadhiya
🙌 1
Sumit Singh
08/08/2023, 1:07 PM@Krutik Gadhiya What the Grafana Dashboard ID you are using ?
Sumit Singh
08/08/2023, 1:08 PMfor me, its only showing last checkpoint duration, flink version used in 1.17
43 Views