I want to monitor the flink app but the problem is...
# troubleshootinga
Aly Ayman
08/11/2024, 5:58 PMI want to monitor the flink app but the problem is only shows as one block , so there is no metric i can see on flink web ui … any recommendation how to monitor it ?
d
D. Draco O'Brien
08/11/2024, 6:04 PMwhat setting do you have for metrics.level in flink-conf.yaml? It can be set to INFO or DEBUG
D. Draco O'Brien
08/11/2024, 6:06 PMalso make sure that metrics.reporter.<reporter-id>.class is set correctly to a class that is suitable to your monitoring system ie JMX or Prometheus etc
a
Aly Ayman
08/11/2024, 6:07 PMhttps://issues.apache.org/jira/browse/FLINK-7286 this jira ticket tells what my exactly problem is
d
D. Draco O'Brien
08/11/2024, 6:09 PMOk, I wonder if somewhere in your application code you could report this using Flink Metric Group API?
a
Aly Ayman
08/11/2024, 6:10 PMThe problem this , I don’t have the sourve code of this app , just the jar
d
D. Draco O'Brien
08/11/2024, 6:14 PMThere is a Kafka Exporter for Prometheus
D. Draco O'Brien
08/11/2024, 6:16 PMOther than that .. you could get a rough estimate from network traffic or view it using something like Wireshark if you can run that in your env.
D. Draco O'Brien
08/11/2024, 6:22 PMActually the best solution if you have full control over the deployment but only have access to the jar would be to leave the operators as-is and introduce a ProcessFunction (or Keyed Process Function) and wrap the source with it. The wrapper function would pass through the records as they are but also update counters/metrics
D. Draco O'Brien
08/11/2024, 6:25 PMTo implement this you just inherit the Keyed/ProcessFunction and implement the processElement method.
Copy code
ctx.forwardElement(element); //to pass downstream without alternationD. Draco O'Brien
08/11/2024, 6:27 PM Copy code
getRuntimeContext().getMetricGroup() // use .counter(..) or .guage(..) methods to report metrics. AFTER the forward.D. Draco O'Brien
08/11/2024, 6:27 PMKeep the implementation of the function very lightweight, and package it in a separate jar.
D. Draco O'Brien
08/11/2024, 6:28 PMTest it carefully in staging. This is your best option I think for now.
D. Draco O'Brien
08/11/2024, 6:32 PMAlso be sure to configure the new operator/process function to be chained if appropriate. Modify the jobs dataflow to include the process function after adding it’s jar to the classpath.
a
Aly Ayman
08/11/2024, 6:34 PMI will give a try tomorrow and come back to you
Aly Ayman
08/11/2024, 6:35 PMThanks Draco
d
D. Draco O'Brien
08/11/2024, 6:35 PMYou’re welcome.
D. Draco O'Brien
08/11/2024, 6:37 PMPrometheus using Kafka Exporter is probably the best choice if it works in your environment otherwise custom Process Function.
a
Aly Ayman
08/11/2024, 6:38 PMOk. I will try kafka exporter first
Aly Ayman
08/12/2024, 11:55 AMHi, I tried to use process function and it worked .. Thanks