How does autoscale work on Flink Kubernetes operat...
# troubleshootingy
Yaroslav Bezruchenko
06/16/2023, 5:49 PMHow does autoscale work on Flink Kubernetes operator 1.5
Copy code
kubernetes.operator.job.autoscaler.enabled: "true"
kubernetes.operator.job.autoscaler.stabilization.interval: "1m"
kubernetes.operator.job.autoscaler.metrics.window: "3m" Copy code
Scaling vertices: Vertex ID a267aa7696f616c525ab17d1bd9850c8 | Parallelism 8 -> 4 | Processing capacity Infinity -> Infinity | Target data rate 8.55 Vertex ID 5570a4732e197e06bd668b2739d98f8c | Parallelism 8 -> 4 | Processing capacity Infinity -> Infinity | Target data rate 8.81 Vertex ID 2963852293169ba90d9d1e7d6308db5c | Parallelism 8 -> 4 | Processing capacity Infinity -> Infinity | Target data rate 0.00 Vertex ID c3c7122991bd9906cb9d1effc69b74a3 | Parallelism 8 -> 6 | Processing capacity 92.21 -> 47.00 | Target data rate 23.36 Vertex ID 456225327bbc50a04e000587e8caa7d1 | Parallelism 8 -> 4 | Processing capacity Infinity -> Infinity | Target data rate 22.74 Vertex ID c49c06ed19ebbb8d7fd89425e1732dc8 | Parallelism 8 -> 4 | Processing capacity Infinity -> Infinity | Target data rate 22.78 Vertex ID 6dc0226b15c44c9c2e1f9ea1a65fd400 | Parallelism 8 -> 4 | Processing capacity Infinity -> Infinity | Target data rate 1.84 Vertex ID 5c4ca2fea30dcf09bf3ee40c495fe808 | Parallelism 8 -> 4 | Processing capacity Infinity -> Infinity | Target data rate 0.00 Vertex ID 4952818ce422a2d9bf045a1179b37754 | Parallelism 8 -> 4 | Processing capacity Infinity -> Infinity | Target data rate 22.40 Vertex ID 47c512bbcb5e27815beb7632e2a43ec4 | Parallelism 8 -> 4 | Processing capacity Infinity -> Infinity | Target data rate 8.81 Vertex ID 7b1f408ee98c276c042c8126a0421765 | Parallelism 8 -> 4 | Processing capacity Infinity -> Infinity | Target data rate 22.71 Vertex ID d412772f7356afb1e854a3e17e6fc99e | Parallelism 8 -> 4 | Processing capacity Infinity -> Infinity | Target data rate 22.30 Vertex ID 02dc93997fd5aa6594c92b9fccde27e2 | Parallelism 8 -> 4 | Processing capacity Infinity -> Infinity | Target data rate 22.23 Vertex ID 3b1890aff09d9152377d394a8c3ad1d5 | Parallelism 8 -> 4 | Processing capacity Infinity -> Infinity | Target data rate 22.31 Vertex ID bf1889e653b6e7d928d2bd2604dca8dd | Parallelism 8 -> 4 | Processing capacity Infinity -> Infinity | Target data rate 19.88 Vertex ID e20cdc086f87b1f78afa7062ae50c3da | Parallelism 8 -> 4 | Processing capacity Infinity -> Infinity | Target data rate 19.93 Vertex ID 6cdc5bb954874d922eaee11a8e7b5dd5 | Parallelism 8 -> 4 | Processing capacity Infinity -> Infinity | Target data rate 0.00 Vertex ID de53745574bc42e95b1357827cc9f260 | Parallelism 8 -> 4 | Processing capacity Infinity -> Infinity | Target data rate 8.77 Vertex ID 1abb23b8bef7c5612f869be2fb3fcf61 | Parallelism 8 -> 4 | Processing capacity Infinity -> Infinity | Target data rate 8.39 Vertex ID 91ae58460ac994d528a45eda171f987a | Parallelism 8 -> 4 | Processing capacity Infinity -> Infinity | Target data rate 22.47 Vertex ID 7cbd5491ae25a563a6d4672f34d0f931 | Parallelism 8 -> 4 | Processing capacity Infinity -> Infinity | Target data rate 8.45 Vertex ID 01c3449094a3cab3b695255ee333decb | Parallelism 8 -> 4 | Processing capacity Infinity -> Infinity | Target data rate 20.10 Vertex ID cbc357ccb763df2852fee8c4fc7d55f2 | Parallelism 8 -> 4 | Processing capacity Infinity -> Infinity | Target data rate 8.50 Vertex ID 7b958a0c17d67bafbb848d71292964cd | Parallelism 8 -> 4 | Processing capacity Infinity -> Infinity | Target data rate 20.02 Vertex ID adc382fc56e42e5a1c3e97cd60017201 | Parallelism 8 -> 4 | Processing capacity Infinity -> Infinity | Target data rate 22.50 Vertex ID 1171dea6747ab509fdaefbe74f7195af | Parallelism 8 -> 4 | Processing capacity Infinity -> Infinity | Target data rate 0.00 Vertex ID 2fa5fc7df17f61cb0e1946288970d799 | Parallelism 8 -> 4 | Processing capacity Infinity -> Infinity | Target data rate 0.00 Vertex ID 79eb943b5d8344895c53163f0bcb07cb | Parallelism 8 -> 4 | Processing capacity Infinity -> Infinity | Target data rate 22.77 Vertex ID 790ed4d19311724271d676c43c6e3949 | Parallelism 8 -> 4 | Processing capacity Infinity -> Infinity | Target data rate 8.73 Vertex ID b1a2a2523a4642215643a6a4e58f0d05 | Parallelism 8 -> 4 | Processing capacity Infinity -> Infinity | Target data rate 0.00 Vertex ID 760b6fbf7de95dce57f3d3f87acab928 | Parallelism 8 -> 4 | Processing capacity Infinity -> Infinity | Target data rate 8.52g
Gyula Fóra
06/19/2023, 3:40 AMBased on these logs, your job is not processing almost any records so the autoscaler gradually scales down to 1
Gyula Fóra
06/19/2023, 3:41 AMThere is a max scale down factor that is by default set around 60% so it won’t scale down more than that in a single step thats why you see this more than once
2 Views