Upgraded the flink k8s operator to `v1 6 0` but the operator Apache Flink #troubleshooting

Upgraded the flink-k8s-operator to `v1.6.0`, but t...

Zhong Chen

07/31/2023, 4:25 PM

Upgraded the flink-k8s-operator to

v1.6.0

, but the operator failed to start.

Zhong Chen

07/31/2023, 4:25 PM

Copy code

2023-07-31 16:18:37,692 i.j.o.p.e.s.i.InformerWrapper  [ERROR] Informer startup error. Operator will be stopped. Informer: <http://flink.apache.org/v1beta1/flinksessionjobs|flink.apache.org/v1beta1/flinksessionjobs>
java.util.concurrent.ExecutionException: java.net.ConnectException: Failed to connect to /172.20.0.1:443
	at java.base/java.util.concurrent.CompletableFuture.reportGet(Unknown Source)
	at java.base/java.util.concurrent.CompletableFuture.get(Unknown Source)
	at io.javaoperatorsdk.operator.processing.event.source.informer.InformerWrapper.start(InformerWrapper.java:85)
	at io.javaoperatorsdk.operator.processing.event.source.informer.InformerManager.lambda$start$0(InformerManager.java:64)
	at io.javaoperatorsdk.operator.api.config.ExecutorServiceManager.lambda$executeAndWaitForAllToComplete$0(ExecutorServiceManager.java:101)
	at java.base/java.util.concurrent.FutureTask.run(Unknown Source)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
	at java.base/java.lang.Thread.run(Unknown Source)
Caused by: java.net.ConnectException: Failed to connect to /172.20.0.1:443
	at okhttp3.internal.connection.RealConnection.connectSocket(RealConnection.kt:297)
	at okhttp3.internal.connection.RealConnection.connect(RealConnection.kt:207)
	at okhttp3.internal.connection.ExchangeFinder.findConnection(ExchangeFinder.kt:226)
	at okhttp3.internal.connection.ExchangeFinder.findHealthyConnection(ExchangeFinder.kt:106)
	at okhttp3.internal.connection.ExchangeFinder.find(ExchangeFinder.kt:74)
	at okhttp3.internal.connection.RealCall.initExchange$okhttp(RealCall.kt:255)
	at okhttp3.internal.connection.ConnectInterceptor.intercept(ConnectInterceptor.kt:32)
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109)
	at okhttp3.internal.cache.CacheInterceptor.intercept(CacheInterceptor.kt:95)
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109)
	at okhttp3.internal.http.BridgeInterceptor.intercept(BridgeInterceptor.kt:83)
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109)
	at okhttp3.internal.http.RetryAndFollowUpInterceptor.intercept(RetryAndFollowUpInterceptor.kt:76)
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109)
	at org.apache.flink.kubernetes.operator.metrics.KubernetesClientMetrics.intercept(KubernetesClientMetrics.java:130)
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109)
	at okhttp3.internal.connection.RealCall.getResponseWithInterceptorChain$okhttp(RealCall.kt:201)
	at okhttp3.internal.connection.RealCall$AsyncCall.run(RealCall.kt:517)
	... 3 more
Caused by: java.net.ConnectException: Connection refused (Connection refused)
	at java.base/java.net.PlainSocketImpl.socketConnect(Native Method)
	at java.base/java.net.AbstractPlainSocketImpl.doConnect(Unknown Source)
	at java.base/java.net.AbstractPlainSocketImpl.connectToAddress(Unknown Source)
	at java.base/java.net.AbstractPlainSocketImpl.connect(Unknown Source)
	at java.base/java.net.SocksSocketImpl.connect(Unknown Source)
	at java.base/java.net.Socket.connect(Unknown Source)
	at okhttp3.internal.platform.Platform.connectSocket(Platform.kt:128)
	at okhttp3.internal.connection.RealConnection.connectSocket(RealConnection.kt:295)
	... 20 more
Exception in thread "main" io.javaoperatorsdk.operator.OperatorException: Error starting operator
	at io.javaoperatorsdk.operator.Operator.start(Operator.java:143)
	at org.apache.flink.kubernetes.operator.FlinkOperator.run(FlinkOperator.java:228)
	at org.apache.flink.kubernetes.operator.FlinkOperator.main(FlinkOperator.java:242)
Caused by: io.javaoperatorsdk.operator.OperatorException: io.javaoperatorsdk.operator.OperatorException: Couldn't start source ControllerResourceEventSource
	at io.javaoperatorsdk.operator.api.config.ExecutorServiceManager.lambda$executeAndWaitForAllToComplete$2(ExecutorServiceManager.java:112)
	at java.base/java.util.ArrayList.forEach(Unknown Source)
	at io.javaoperatorsdk.operator.api.config.ExecutorServiceManager.executeAndWaitForAllToComplete(ExecutorServiceManager.java:107)
	at io.javaoperatorsdk.operator.api.config.ExecutorServiceManager.boundedExecuteAndWaitForAllToComplete(ExecutorServiceManager.java:87)
	at io.javaoperatorsdk.operator.ControllerManager.start(ControllerManager.java:36)
	at io.javaoperatorsdk.operator.Operator.start(Operator.java:138)
	... 2 more
Caused by: io.javaoperatorsdk.operator.OperatorException: Couldn't start source ControllerResourceEventSource
	at io.javaoperatorsdk.operator.processing.event.EventSourceManager.startEventSource(EventSourceManager.java:125)
	at io.javaoperatorsdk.operator.processing.event.EventSourceManager.start(EventSourceManager.java:66)
	at io.javaoperatorsdk.operator.processing.Controller.start(Controller.java:327)
	at io.javaoperatorsdk.operator.ControllerManager.lambda$start$0(ControllerManager.java:37)
	at io.javaoperatorsdk.operator.api.config.ExecutorServiceManager.lambda$executeAndWaitForAllToComplete$0(ExecutorServiceManager.java:101)
	at java.base/java.util.concurrent.FutureTask.run(Unknown Source)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
	at java.base/java.lang.Thread.run(Unknown Source)
Caused by: io.javaoperatorsdk.operator.OperatorException: io.javaoperatorsdk.operator.OperatorException: Couldn't start informer for <http://flinksessionjobs.flink.apache.org/v1beta1|flinksessionjobs.flink.apache.org/v1beta1> resources
	at io.javaoperatorsdk.operator.api.config.ExecutorServiceManager.lambda$executeAndWaitForAllToComplete$2(ExecutorServiceManager.java:112)
	at java.base/java.util.ArrayList.forEach(Unknown Source)
	at io.javaoperatorsdk.operator.api.config.ExecutorServiceManager.executeAndWaitForAllToComplete(ExecutorServiceManager.java:107)
	at io.javaoperatorsdk.operator.api.config.ExecutorServiceManager.boundedExecuteAndWaitForAllToComplete(ExecutorServiceManager.java:87)
	at io.javaoperatorsdk.operator.processing.event.source.informer.InformerManager.start(InformerManager.java:62)
	at io.javaoperatorsdk.operator.processing.event.source.informer.ManagedInformerEventSource.start(ManagedInformerEventSource.java:79)
	at io.javaoperatorsdk.operator.processing.event.source.controller.ControllerResourceEventSource.start(ControllerResourceEventSource.java:61)
	at io.javaoperatorsdk.operator.processing.event.NamedEventSource.start(NamedEventSource.java:27)
	at io.javaoperatorsdk.operator.processing.event.EventSourceManager.startEventSource(EventSourceManager.java:120)
	... 8 more
Caused by: io.javaoperatorsdk.operator.OperatorException: Couldn't start informer for <http://flinksessionjobs.flink.apache.org/v1beta1|flinksessionjobs.flink.apache.org/v1beta1> resources
	at io.javaoperatorsdk.operator.processing.event.source.informer.InformerWrapper.start(InformerWrapper.java:108)
	at io.javaoperatorsdk.operator.processing.event.source.informer.InformerManager.lambda$start$0(InformerManager.java:64)
	... 5 more
Caused by: io.javaoperatorsdk.operator.OperatorException: java.util.concurrent.ExecutionException: java.net.ConnectException: Failed to connect to /172.20.0.1:443
	at io.javaoperatorsdk.operator.processing.event.source.informer.InformerWrapper.start(InformerWrapper.java:92)
	... 6 more
Caused by: java.util.concurrent.ExecutionException: java.net.ConnectException: Failed to connect to /172.20.0.1:443
	at java.base/java.util.concurrent.CompletableFuture.reportGet(Unknown Source)
	at java.base/java.util.concurrent.CompletableFuture.get(Unknown Source)
	at io.javaoperatorsdk.operator.processing.event.source.informer.InformerWrapper.start(InformerWrapper.java:85)
	... 6 more
Caused by: java.net.ConnectException: Failed to connect to /172.20.0.1:443
	at okhttp3.internal.connection.RealConnection.connectSocket(RealConnection.kt:297)
	at okhttp3.internal.connection.RealConnection.connect(RealConnection.kt:207)
	at okhttp3.internal.connection.ExchangeFinder.findConnection(ExchangeFinder.kt:226)
	at okhttp3.internal.connection.ExchangeFinder.findHealthyConnection(ExchangeFinder.kt:106)
	at okhttp3.internal.connection.ExchangeFinder.find(ExchangeFinder.kt:74)
	at okhttp3.internal.connection.RealCall.initExchange$okhttp(RealCall.kt:255)
	at okhttp3.internal.connection.ConnectInterceptor.intercept(ConnectInterceptor.kt:32)
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109)
	at okhttp3.internal.cache.CacheInterceptor.intercept(CacheInterceptor.kt:95)
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109)
	at okhttp3.internal.http.BridgeInterceptor.intercept(BridgeInterceptor.kt:83)
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109)
	at okhttp3.internal.http.RetryAndFollowUpInterceptor.intercept(RetryAndFollowUpInterceptor.kt:76)
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109)
	at org.apache.flink.kubernetes.operator.metrics.KubernetesClientMetrics.intercept(KubernetesClientMetrics.java:130)
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109)
	at okhttp3.internal.connection.RealCall.getResponseWithInterceptorChain$okhttp(RealCall.kt:201)
	at okhttp3.internal.connection.RealCall$AsyncCall.run(RealCall.kt:517)
	... 3 more
Caused by: java.net.ConnectException: Connection refused (Connection refused)
	at java.base/java.net.PlainSocketImpl.socketConnect(Native Method)
	at java.base/java.net.AbstractPlainSocketImpl.doConnect(Unknown Source)
	at java.base/java.net.AbstractPlainSocketImpl.connectToAddress(Unknown Source)
	at java.base/java.net.AbstractPlainSocketImpl.connect(Unknown Source)
	at java.base/java.net.SocksSocketImpl.connect(Unknown Source)
	at java.base/java.net.Socket.connect(Unknown Source)
	at okhttp3.internal.platform.Platform.connectSocket(Platform.kt:128)
	at okhttp3.internal.connection.RealConnection.connectSocket(RealConnection.kt:295)
	... 20 more

Zhong Chen

07/31/2023, 4:25 PM

Is this a permission issue?

Copy code

Caused by: java.net.ConnectException: Failed to connect to /172.20.0.1:443

Zhong Chen

07/31/2023, 4:48 PM

Copy code

[ERROR] Informer startup error. Operator will be stopped. Informer: <http://flink.apache.org/v1beta1/flinksessionjobs|flink.apache.org/v1beta1/flinksessionjobs>
java.util.concurrent.ExecutionException: io.fabric8.kubernetes.client.KubernetesClientException: Failure executing: GET at: <https://172.20.0.1:443/apis/flink.apache.org/v1beta1/flinksessionjobs?resourceVersion=0>. Message: Not Found.
	at java.base/java.util.concurrent.CompletableFuture.reportGet(Unknown Source)
	at java.base/java.util.concurrent.CompletableFuture.get(Unknown Source)
	at io.javaoperatorsdk.operator.processing.event.source.informer.InformerWrapper.start(InformerWrapper.java:85)
	at io.javaoperatorsdk.operator.processing.event.source.informer.InformerManager.lambda$start$0(InformerManager.java:64)
	at io.javaoperatorsdk.operator.api.config.ExecutorServiceManager.lambda$executeAndWaitForAllToComplete$0(ExecutorServiceManager.java:101)
	at java.base/java.util.concurrent.FutureTask.run(Unknown Source)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
	at java.base/java.lang.Thread.run(Unknown Source)
Caused by: io.fabric8.kubernetes.client.KubernetesClientException: Failure executing: GET at: <https://172.20.0.1:443/apis/flink.apache.org/v1beta1/flinksessionjobs?resourceVersion=0>. Message: Not Found.
	at io.fabric8.kubernetes.client.dsl.internal.OperationSupport.requestFailure(OperationSupport.java:701)
	at io.fabric8.kubernetes.client.dsl.internal.OperationSupport.requestFailure(OperationSupport.java:681)
	at io.fabric8.kubernetes.client.dsl.internal.OperationSupport.assertResponseCode(OperationSupport.java:630)
	at io.fabric8.kubernetes.client.dsl.internal.OperationSupport.lambda$handleResponse$0(OperationSupport.java:591)
	at java.base/java.util.concurrent.CompletableFuture$UniApply.tryFire(Unknown Source)
	at java.base/java.util.concurrent.CompletableFuture.postComplete(Unknown Source)
	at java.base/java.util.concurrent.CompletableFuture.complete(Unknown Source)
	at io.fabric8.kubernetes.client.http.StandardHttpClient.lambda$completeOrCancel$5(StandardHttpClient.java:120)
	at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(Unknown Source)
	at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(Unknown Source)
	at java.base/java.util.concurrent.CompletableFuture.postComplete(Unknown Source)
	at java.base/java.util.concurrent.CompletableFuture.complete(Unknown Source)
	at io.fabric8.kubernetes.client.http.ByteArrayBodyHandler.onBodyDone(ByteArrayBodyHandler.java:52)
	at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(Unknown Source)
	at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(Unknown Source)
	at java.base/java.util.concurrent.CompletableFuture.postComplete(Unknown Source)
	at java.base/java.util.concurrent.CompletableFuture.complete(Unknown Source)
	at io.fabric8.kubernetes.client.okhttp.OkHttpClientImpl$OkHttpAsyncBody.doConsume(OkHttpClientImpl.java:135)
	... 3 more

Zhong Chen

07/31/2023, 4:49 PM

Copy code

2023-07-31 16:33:45,444 o.a.f.k.o.FlinkOperator        [INFO ] Configuring operator to select custom resources with the null labels.
2023-07-31 16:33:45,491 i.j.o.Operator                 [INFO ] Registered reconciler: 'flinkdeploymentcontroller' for resource: 'class org.apache.flink.kubernetes.operator.api.FlinkDeployment' for namespace(s): [all namespaces]
2023-07-31 16:33:45,500 Default ConfigurationService implementation [WARN ] Configuration for reconciler 'flinksessionjobcontroller' was not found. Known reconcilers: None.
2023-07-31 16:33:45,501 Default ConfigurationService implementation [INFO ] Created configuration for reconciler org.apache.flink.kubernetes.operator.controller.FlinkSessionJobController with name flinksessionjobcontroller
2023-07-31 16:33:45,501 o.a.f.k.o.FlinkOperator        [INFO ] Configuring operator to watch the following namespaces: [JOSDK_ALL_NAMESPACES].
2023-07-31 16:33:45,501 o.a.f.k.o.FlinkOperator        [INFO ] Configuring operator to select custom resources with the null labels.
2023-07-31 16:33:45,505 i.j.o.Operator                 [INFO ] Registered reconciler: 'flinksessionjobcontroller' for resource: 'class org.apache.flink.kubernetes.operator.api.FlinkSessionJob' for namespace(s): [all namespaces]
2023-07-31 16:33:45,508 i.j.o.Operator                 [INFO ] Operator SDK 4.3.0 (commit: b410c65) built on 2023-03-31T08:11:47.000+0000 starting...
2023-07-31 16:33:45,508 i.j.o.Operator                 [INFO ] Client version: 6.5.1
2023-07-31 16:33:45,513 i.j.o.p.Controller             [INFO ] Starting 'flinksessionjobcontroller' controller for reconciler: org.apache.flink.kubernetes.operator.controller.FlinkSessionJobController, resource: org.apache.flink.kubernetes.operator.api.FlinkSessionJob
2023-07-31 16:33:45,512 i.j.o.p.Controller             [INFO ] Starting 'flinkdeploymentcontroller' controller for reconciler: org.apache.flink.kubernetes.operator.controller.FlinkDeploymentController, resource: org.apache.flink.kubernetes.operator.api.FlinkDeployment
2023-07-31 16:33:45,586 i.f.k.c.d.i.VersionUsageUtils  [WARN ] The client is using resource type 'flinksessionjobs' with unstable version 'v1beta1'
2023-07-31 16:33:45,587 i.f.k.c.d.i.VersionUsageUtils  [WARN ] The client is using resource type 'flinkdeployments' with unstable version 'v1beta1'

Zhong Chen

07/31/2023, 7:11 PM

found the issue. looks like the

crds

are missing in the cluster.

Zhong Chen

07/31/2023, 7:12 PM

In 1.5, the doc recommends splitting the argo app definition into two parts, but in 1.6, it consolidates two apps into a single one.

Zhong Chen

07/31/2023, 7:12 PM

1.5: https://nightlies.apache.org/flink/flink-kubernetes-operator-docs-release-1.5/docs/operations/helm/#working-with-argo-cd 1.6: https://nightlies.apache.org/flink/flink-kubernetes-operator-docs-release-1.6/docs/operations/helm/#working-with-argo-cd

Zhong Chen

07/31/2023, 7:29 PM

and I forgot to remove

skipCrds: true

in my argo app definition.

2 Views

Open in Slack

Previous Next