I m trying kube autoscaler getting Some errors and warn logs Apache Flink #troubleshooting

I'm trying kube autoscaler, getting Some errors an...

Gaurav Miglani

02/02/2023, 6:33 PM

I'm trying kube autoscaler, getting Some errors and warn logs, also job is not able to scale somehow, am i missing something

Gaurav Miglani

02/02/2023, 6:33 PM

Copy code

2023-02-02 18:31:21,598 o.a.f.k.o.a.m.ScalingMetrics   [ERROR][default/<job_name>] Cannot compute source target data rate without numRecordsInPerSecond and pendingRecords (lag) metric for cbc357ccb763df2852fee8c4fc7d55f2.
2023-02-02 18:31:21,603 o.a.f.k.o.a.JobVertexScaler    [WARN ][default/<job_name>] Target data rate is not available for cbc357ccb763df2852fee8c4fc7d55f2, cannot compute new parallelism
2023-02-02 18:31:21,603 o.a.f.k.o.a.JobVertexScaler    [WARN ][default/<job_name>] Target data rate is not available for 0b1bf63425181a3400bc50b290a60d8f, cannot compute new parallelism
2023-02-02 18:31:21,603 o.a.f.k.o.a.JobVertexScaler    [WARN ][default/<job_name>] True processing rate is not available for da67a5f4d5454f96aea7db4adede3387, cannot compute new parallelism
2023-02-02 18:31:21,603 o.a.f.k.o.a.JobVertexScaler    [WARN ][default/<job_name>] Target data rate is not available for 98fb832d4cfc2f49f0d9cac59391f442, cannot compute new parallelism
2023-02-02 18:31:21,603 o.a.f.k.o.a.JobVertexScaler    [WARN ][default/<job_name>] Target data rate is not available for 6867c353ece92b197202ac2725ffb603, cannot compute new parallelism
2023-02-02 18:31:21,603 o.a.f.k.o.a.JobVertexScaler    [WARN ][default/<job_name>] True processing rate is not available for e3552248be1282b297b9879c6301fb94, cannot compute new parallelism
2023-02-02 18:31:21,604 o.a.f.k.o.a.JobVertexScaler    [WARN ][default/<job_name>] Target data rate is not available for 40b66110ab7fa0470145727e436613cd, cannot compute new parallelism

Gaurav Miglani

02/02/2023, 6:34 PM

job is reading data from kafka, doing window aggregations, nothing more, pure flink sql job

Gaurav Miglani

02/02/2023, 6:35 PM

Copy code

2023-02-02 18:31:36,772 o.a.f.k.o.a.JobVertexScaler    [WARN ][default/<job_name>] True processing rate is not available for da67a5f4d5454f96aea7db4adede3387, cannot compute new parallelism
2023-02-02 18:31:36,772 o.a.f.k.o.a.JobVertexScaler    [WARN ][default/<job_name>] True processing rate is not available for e3552248be1282b297b9879c6301fb94, cannot compute new parallelism
2023-02-02 18:31:36,825 o.a.f.k.o.l.AuditUtils         [INFO ][default/<job_name>] >>> Event  | Info    | SCALINGREPORT   | Scaling vertices: Vertex ID cbc357ccb763df2852fee8c4fc7d55f2 | Parallelism 16 -> 150 | Processing capacity 287482.08 -> 41076019.00 | Target data rate 112250.50 Vertex ID 0b1bf63425181a3400bc50b290a60d8f | Parallelism 16 -> 180 | Processing capacity 926009.51 -> 34343485.00 | Target data rate 93852.17 Vertex ID 98fb832d4cfc2f49f0d9cac59391f442 | Parallelism 16 -> 30 | Processing capacity 28822.22 -> 39022.00 | Target data rate 106.64 Vertex ID 6867c353ece92b197202ac2725ffb603 | Parallelism 16 -> 180 | Processing capacity 198472.77 -> 34343485.00 | Target data rate 93852.17 Vertex ID 40b66110ab7fa0470145727e436613cd | Parallelism 16 -> 180 | Processing capacity 185610.82 -> 34343485.00 | Target data rate 93852.17

Gaurav Miglani

02/02/2023, 6:36 PM

service and tm pods are getting restarted, but count of TM pods not increasing 🤔

Gaurav Miglani

02/02/2023, 6:38 PM

flink lag and input metrics are exposed to datadog and i can see them as well

Open in Slack

Previous Next