I have 4 task manager each one is on separate mach...
# troubleshootinga
Aly Ayman
08/26/2024, 5:49 AMI have 4 task manager each one is on separate machine and one job manager
All task managers are reporting metrics to job manager as i see in web ui
Except one task manager so if i have custom metrics like metric event it will be shown in graph for all tasks that runs except for this task manager
All the configs is the same , we usually copy paste the config
a
Alexander Blackhall
08/26/2024, 6:05 AMHave you confirmed tasks are being sent to slots on that taskmanager and it is actually doing work?
a
Aly Ayman
08/26/2024, 6:18 AMYess this machine do the most of work
👍 1
Aly Ayman
08/26/2024, 6:18 AMAm using flink v1.12
a
Alexander Blackhall
08/26/2024, 7:06 AMOne thing we can say is that’s an old version, but that’s probably not the source of your problem. I wish you luck 🙇♂️
k
Ken Krugler
08/26/2024, 1:42 PMSeems very unlikely this is a Flink bug, but I assume you checked the logs on that one server, as well as the JM logs for any unusual messages.
d
D. Draco O'Brien
08/26/2024, 2:32 PMMake sure everything is on same version of Flink v1.12, how are CPU, Memory and Diskspace doing?
a
Aly Ayman
08/26/2024, 2:32 PMEven it didn’t report memory’s metrics .
Yes nothing unordinary in the logs
d
D. Draco O'Brien
08/26/2024, 2:33 PMok, can you restart the task manager service and show the logs from that point onward?
D. Draco O'Brien
08/26/2024, 2:33 PMAlso whats your deployment again? you are on k8s? or docker?
a
Aly Ayman
08/26/2024, 2:34 PMStandalone cluster
Aly Ayman
08/26/2024, 2:34 PMI restarted many times and checked the logs . Nothing unordinary
d
D. Draco O'Brien
08/26/2024, 2:37 PMok, but do you see anything related to metrics? sending_metrics or reporting_metrics?
D. Draco O'Brien
08/26/2024, 2:37 PMwith proper log levels it should at least be reporting the attempt to send the metrics right?
D. Draco O'Brien
08/26/2024, 2:38 PMIf the taskmanager is setup to send metrics then it will log the attempt to send even if it never reaches the job manager. At least I think thats how it would work
D. Draco O'Brien
08/26/2024, 2:39 PMor just anything in the taskmanager logs that contains the words metrics or metric?
a
Aly Ayman
08/26/2024, 2:41 PMIf you meant “no metrics will be reported” .. it is the same for all logs in task managers
d
D. Draco O'Brien
08/26/2024, 2:45 PMok .. well …
D. Draco O'Brien
08/26/2024, 2:48 PMfrom those logs it does not look like metrics are properly configured.
a
Aly Ayman
08/26/2024, 2:50 PMYes but it is the same for task managers that worked right
d
D. Draco O'Brien
08/26/2024, 2:50 PMBasically if it attempts to send metrics it will look like
Copy code
DEBUG org.apache.flink.runtime.metrics.MetricRegistryImpl - Reporting metrics to '<akka.tcp://flink@jobmanager-host>:port/user/taskmanager_0'.D. Draco O'Brien
08/26/2024, 2:50 PMand that’s even if it never makes it to the JobManager
D. Draco O'Brien
08/26/2024, 2:50 PMYou should be seeing this if its properly configured.
D. Draco O'Brien
08/26/2024, 2:51 PMthere also might be some metric gathering activities like INFO org.apache.flink.metrics.groups.TaskManagerMetricGroup - Metric ‘metric.name’ updated to ‘value’.
``````
D. Draco O'Brien
08/26/2024, 2:53 PMYou might not see acks but you should see something like
Copy code
DEBUG org.apache.flink.runtime.jobmaster.JobMaster - Received metric report from TaskManager with ID=taskmanager-id.D. Draco O'Brien
08/26/2024, 2:53 PMDo you see these for all accept one task-manager in the logs? Its a debug level message.