What can be the cause of BUSY_TIME_MS metrics not ...
# troubleshootingo
Otto Remse
10/13/2023, 1:06 PMWhat can be the cause of BUSY_TIME_MS metrics not existing on vertices? We sporadically encounter issues where either none of the tasks have that metric or some of them lack them, which makes the autoscaling feature in the kubernetes operator useless. All tasks in the dashboard have "N/A" on busy and backpressured. (running flink 1.17 on operator 1.6 with a flink-sql job)
Otto Remse
10/13/2023, 1:19 PMThe only thing I can see in the logs is
Copy code
No hostname could be resolved for the IP address xx.xx.xx.xx, using IP address as host name. Local input split assignment (such as for HDFS files) may be impacted.g
Gyula Fóra
10/16/2023, 5:40 AMSome source connectors (that use the legacy source api) do not have it
Gyula Fóra
10/16/2023, 5:40 AMBut otherwise non source vertices should all have it
o
Otto Remse
10/16/2023, 7:16 AMThe issue is that they do have the metrics at one deployment, but then the next time they might be gone. Sporadically.
g
Gyula Fóra
10/16/2023, 9:47 AMYou may get a metric error initially as not all metrics are visible when we query the first time but eventually they should exist
o
Otto Remse
10/17/2023, 7:25 AMThis is 15 hours into the run and still not getting any busy time metrics on any of the operators. Very unclear what causes this as it seems to only be on some of the deployments (with the exact same job code and config) Sources are kafka upsert