https://flink.apache.org/ logo
Join Slack
Powered by
Hey guys! I'm deploying the Flink K8s Operator in ...
# troubleshooting
s
Hey guys! I'm deploying the Flink K8s Operator in a cluster but it gets stopped due to the following error:
Copy code
io.javaoperatorsdk.operator.OperatorException: Couldn't start informer for <http://flinksessionjobs.flink.apache.org/v1beta1|flinksessionjobs.flink.apache.org/v1beta1> resources
In reply you can see the complete stack strace. Does anyone know how to solve this problem?
Copy code
Exception in thread "main" io.javaoperatorsdk.operator.OperatorException: Error starting operator
	at io.javaoperatorsdk.operator.Operator.start(Operator.java:143)
	at org.apache.flink.kubernetes.operator.FlinkOperator.run(FlinkOperator.java:228)
	at org.apache.flink.kubernetes.operator.FlinkOperator.main(FlinkOperator.java:242)
Caused by: io.javaoperatorsdk.operator.OperatorException: io.javaoperatorsdk.operator.OperatorException: Couldn't start source ControllerResourceEventSource
	at io.javaoperatorsdk.operator.api.config.ExecutorServiceManager.lambda$executeAndWaitForAllToComplete$2(ExecutorServiceManager.java:112)
	at java.base/java.util.ArrayList.forEach(Unknown Source)
	at io.javaoperatorsdk.operator.api.config.ExecutorServiceManager.executeAndWaitForAllToComplete(ExecutorServiceManager.java:107)
	at io.javaoperatorsdk.operator.api.config.ExecutorServiceManager.boundedExecuteAndWaitForAllToComplete(ExecutorServiceManager.java:87)
	at io.javaoperatorsdk.operator.ControllerManager.start(ControllerManager.java:36)
	at io.javaoperatorsdk.operator.Operator.start(Operator.java:138)
	... 2 more
Caused by: io.javaoperatorsdk.operator.OperatorException: Couldn't start source ControllerResourceEventSource
	at io.javaoperatorsdk.operator.processing.event.EventSourceManager.startEventSource(EventSourceManager.java:125)
	at io.javaoperatorsdk.operator.processing.event.EventSourceManager.start(EventSourceManager.java:66)
	at io.javaoperatorsdk.operator.processing.Controller.start(Controller.java:327)
	at io.javaoperatorsdk.operator.ControllerManager.lambda$start$0(ControllerManager.java:37)
	at io.javaoperatorsdk.operator.api.config.ExecutorServiceManager.lambda$executeAndWaitForAllToComplete$0(ExecutorServiceManager.java:101)
	at java.base/java.util.concurrent.FutureTask.run(Unknown Source)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
	at java.base/java.lang.Thread.run(Unknown Source)
Caused by: io.javaoperatorsdk.operator.OperatorException: io.javaoperatorsdk.operator.OperatorException: Couldn't start informer for <http://flinksessionjobs.flink.apache.org/v1beta1|flinksessionjobs.flink.apache.org/v1beta1> resources
	at io.javaoperatorsdk.operator.api.config.ExecutorServiceManager.lambda$executeAndWaitForAllToComplete$2(ExecutorServiceManager.java:112)
	at java.base/java.util.ArrayList.forEach(Unknown Source)
	at io.javaoperatorsdk.operator.api.config.ExecutorServiceManager.executeAndWaitForAllToComplete(ExecutorServiceManager.java:107)
	at io.javaoperatorsdk.operator.api.config.ExecutorServiceManager.boundedExecuteAndWaitForAllToComplete(ExecutorServiceManager.java:87)
	at io.javaoperatorsdk.operator.processing.event.source.informer.InformerManager.start(InformerManager.java:62)
	at io.javaoperatorsdk.operator.processing.event.source.informer.ManagedInformerEventSource.start(ManagedInformerEventSource.java:79)
	at io.javaoperatorsdk.operator.processing.event.source.controller.ControllerResourceEventSource.start(ControllerResourceEventSource.java:61)
	at io.javaoperatorsdk.operator.processing.event.NamedEventSource.start(NamedEventSource.java:27)
	at io.javaoperatorsdk.operator.processing.event.EventSourceManager.startEventSource(EventSourceManager.java:120)
	... 8 more
Caused by: io.javaoperatorsdk.operator.OperatorException: Couldn't start informer for <http://flinksessionjobs.flink.apache.org/v1beta1|flinksessionjobs.flink.apache.org/v1beta1> resources
	at io.javaoperatorsdk.operator.processing.event.source.informer.InformerWrapper.start(InformerWrapper.java:108)
	at io.javaoperatorsdk.operator.processing.event.source.informer.InformerManager.lambda$start$0(InformerManager.java:64)
	... 5 more
Caused by: io.javaoperatorsdk.operator.OperatorException: java.util.concurrent.ExecutionException: java.net.SocketTimeoutException: connect timed out
	at io.javaoperatorsdk.operator.processing.event.source.informer.InformerWrapper.start(InformerWrapper.java:92)
	... 6 more
Caused by: java.util.concurrent.ExecutionException: java.net.SocketTimeoutException: connect timed out
	at java.base/java.util.concurrent.CompletableFuture.reportGet(Unknown Source)
	at java.base/java.util.concurrent.CompletableFuture.get(Unknown Source)
	at io.javaoperatorsdk.operator.processing.event.source.informer.InformerWrapper.start(InformerWrapper.java:85)
	... 6 more
Caused by: java.net.SocketTimeoutException: connect timed out
	at java.base/java.net.PlainSocketImpl.socketConnect(Native Method)
	at java.base/java.net.AbstractPlainSocketImpl.doConnect(Unknown Source)
	at java.base/java.net.AbstractPlainSocketImpl.connectToAddress(Unknown Source)
	at java.base/java.net.AbstractPlainSocketImpl.connect(Unknown Source)
	at java.base/java.net.SocksSocketImpl.connect(Unknown Source)
	at java.base/java.net.Socket.connect(Unknown Source)
	at okhttp3.internal.platform.Platform.connectSocket(Platform.kt:128)
	at okhttp3.internal.connection.RealConnection.connectSocket(RealConnection.kt:295)
	at okhttp3.internal.connection.RealConnection.connect(RealConnection.kt:207)
	at okhttp3.internal.connection.ExchangeFinder.findConnection(ExchangeFinder.kt:226)
	at okhttp3.internal.connection.ExchangeFinder.findHealthyConnection(ExchangeFinder.kt:106)
	at okhttp3.internal.connection.ExchangeFinder.find(ExchangeFinder.kt:74)
	at okhttp3.internal.connection.RealCall.initExchange$okhttp(RealCall.kt:255)
	at okhttp3.internal.connection.ConnectInterceptor.intercept(ConnectInterceptor.kt:32)
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109)
	at okhttp3.internal.cache.CacheInterceptor.intercept(CacheInterceptor.kt:95)
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109)
	at okhttp3.internal.http.BridgeInterceptor.intercept(BridgeInterceptor.kt:83)
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109)
	at okhttp3.internal.http.RetryAndFollowUpInterceptor.intercept(RetryAndFollowUpInterceptor.kt:76)
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109)
	at org.apache.flink.kubernetes.operator.metrics.KubernetesClientMetrics.intercept(KubernetesClientMetrics.java:130)
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109)
	at okhttp3.internal.connection.RealCall.getResponseWithInterceptorChain$okhttp(RealCall.kt:201)
	at okhttp3.internal.connection.RealCall$AsyncCall.run(RealCall.kt:517)
	... 3 more
Within the logs I can also see...
Copy code
{
  "instant": {
    "epochSecond": 1694080911,
    "nanoOfSecond": 710000000
  },
  "thread": "OkHttp <https://100.64.0.1/>...",
  "level": "ERROR",
  "loggerName": "io.javaoperatorsdk.operator.api.config.ConfigurationService",
  "message": "Fatal error in informer: <http://flink.apache.org/v1beta1/namespaces/platform/flinkdeployments|flink.apache.org/v1beta1/namespaces/platform/flinkdeployments>. Stopping the operator",
  "thrown": {
    "commonElementCount": 0,
    "localizedMessage": "io.fabric8.kubernetes.client.KubernetesClientException: An error has occurred.",
    "message": "io.fabric8.kubernetes.client.KubernetesClientException: An error has occurred.",
    "name": "java.util.concurrent.CompletionException",
    "cause": {
      "commonElementCount": 22,
      "localizedMessage": "An error has occurred.",
      "message": "An error has occurred.",
      "name": "io.fabric8.kubernetes.client.KubernetesClientException",
      "cause": {
        "commonElementCount": 3,
        "localizedMessage": "connect timed out",
        "message": "connect timed out",
        "name": "java.net.SocketTimeoutException",
        "extendedStackTrace": [
          {
            "class": "java.net.PlainSocketImpl",
            "method": "socketConnect",
            "line": -2,
            "exact": false,
            "location": "?",
            "version": "?"
          },
          {
            "class": "java.net.AbstractPlainSocketImpl",
            "method": "doConnect",
            "line": -1,
            "exact": false,
            "location": "?",
            "version": "?"
          },
          {
            "class": "java.net.AbstractPlainSocketImpl",
            "method": "connectToAddress",
            "line": -1,
            "exact": false,
            "location": "?",
            "version": "?"
          },
          {
            "class": "java.net.AbstractPlainSocketImpl",
            "method": "connect",
            "line": -1,
            "exact": false,
            "location": "?",
            "version": "?"
          },
          {
            "class": "java.net.SocksSocketImpl",
            "method": "connect",
            "line": -1,
            "exact": false,
            "location": "?",
            "version": "?"
          },
          {
            "class": "java.net.Socket",
            "method": "connect",
            "line": -1,
            "exact": false,
            "location": "?",
            "version": "?"
          },
          {
            "class": "okhttp3.internal.platform.Platform",
            "method": "connectSocket",
            "file": "Platform.kt",
            "line": 128,
            "exact": false,
            "location": "flink-kubernetes-operator-1.5.0-shaded.jar",
            "version": "1.5.0"
          },
          {
            "class": "okhttp3.internal.connection.RealConnection",
            "method": "connectSocket",
            "file": "RealConnection.kt",
            "line": 295,
            "exact": false,
            "location": "flink-kubernetes-operator-1.5.0-shaded.jar",
            "version": "1.5.0"
          },
          {
            "class": "okhttp3.internal.connection.RealConnection",
            "method": "connect",
            "file": "RealConnection.kt",
            "line": 207,
            "exact": false,
            "location": "flink-kubernetes-operator-1.5.0-shaded.jar",
            "version": "1.5.0"
          },
          {
            "class": "okhttp3.internal.connection.ExchangeFinder",
            "method": "findConnection",
            "file": "ExchangeFinder.kt",
            "line": 226,
            "exact": false,
            "location": "flink-kubernetes-operator-1.5.0-shaded.jar",
            "version": "1.5.0"
          },
          {
            "class": "okhttp3.internal.connection.ExchangeFinder",
            "method": "findHealthyConnection",
            "file": "ExchangeFinder.kt",
            "line": 106,
            "exact": false,
            "location": "flink-kubernetes-operator-1.5.0-shaded.jar",
            "version": "1.5.0"
          },
          {
            "class": "okhttp3.internal.connection.ExchangeFinder",
            "method": "find",
            "file": "ExchangeFinder.kt",
            "line": 74,
            "exact": false,
            "location": "flink-kubernetes-operator-1.5.0-shaded.jar",
            "version": "1.5.0"
          },
          {
            "class": "okhttp3.internal.connection.RealCall",
            "method": "initExchange$okhttp",
            "file": "RealCall.kt",
            "line": 255,
            "exact": false,
            "location": "flink-kubernetes-operator-1.5.0-shaded.jar",
            "version": "1.5.0"
          },
          {
            "class": "okhttp3.internal.connection.ConnectInterceptor",
            "method": "intercept",
            "file": "ConnectInterceptor.kt",
            "line": 32,
            "exact": false,
            "location": "flink-kubernetes-operator-1.5.0-shaded.jar",
            "version": "1.5.0"
          },
          {
            "class": "okhttp3.internal.http.RealInterceptorChain",
            "method": "proceed",
            "file": "RealInterceptorChain.kt",
            "line": 109,
            "exact": false,
            "location": "flink-kubernetes-operator-1.5.0-shaded.jar",
            "version": "1.5.0"
          },
          {
            "class": "okhttp3.internal.cache.CacheInterceptor",
            "method": "intercept",
            "file": "CacheInterceptor.kt",
            "line": 95,
            "exact": false,
            "location": "flink-kubernetes-operator-1.5.0-shaded.jar",
            "version": "1.5.0"
          },
          {
            "class": "okhttp3.internal.http.RealInterceptorChain",
            "method": "proceed",
            "file": "RealInterceptorChain.kt",
            "line": 109,
            "exact": false,
            "location": "flink-kubernetes-operator-1.5.0-shaded.jar",
            "version": "1.5.0"
          },
          {
            "class": "okhttp3.internal.http.BridgeInterceptor",
            "method": "intercept",
            "file": "BridgeInterceptor.kt",
            "line": 83,
            "exact": false,
            "location": "flink-kubernetes-operator-1.5.0-shaded.jar",
            "version": "1.5.0"
          },
          {
            "class": "okhttp3.internal.http.RealInterceptorChain",
            "method": "proceed",
            "file": "RealInterceptorChain.kt",
            "line": 109,
            "exact": false,
            "location": "flink-kubernetes-operator-1.5.0-shaded.jar",
            "version": "1.5.0"
          },
          {
            "class": "okhttp3.internal.http.RetryAndFollowUpInterceptor",
            "method": "intercept",
            "file": "RetryAndFollowUpInterceptor.kt",
            "line": 76,
            "exact": false,
            "location": "flink-kubernetes-operator-1.5.0-shaded.jar",
            "version": "1.5.0"
          },
          {
            "class": "okhttp3.internal.http.RealInterceptorChain",
            "method": "proceed",
            "file": "RealInterceptorChain.kt",
            "line": 109,
            "exact": false,
            "location": "flink-kubernetes-operator-1.5.0-shaded.jar",
            "version": "1.5.0"
          },
          {
            "class": "org.apache.flink.kubernetes.operator.metrics.KubernetesClientMetrics",
            "method": "intercept",
            "file": "KubernetesClientMetrics.java",
            "line": 130,
            "exact": false,
            "location": "flink-kubernetes-operator-1.5.0-shaded.jar",
            "version": "1.5.0"
          },
          {
            "class": "okhttp3.internal.http.RealInterceptorChain",
            "method": "proceed",
            "file": "RealInterceptorChain.kt",
            "line": 109,
            "exact": false,
            "location": "flink-kubernetes-operator-1.5.0-shaded.jar",
            "version": "1.5.0"
          },
          {
            "class": "okhttp3.internal.connection.RealCall",
            "method": "getResponseWithInterceptorChain$okhttp",
            "file": "RealCall.kt",
            "line": 201,
            "exact": false,
            "location": "flink-kubernetes-operator-1.5.0-shaded.jar",
            "version": "1.5.0"
          },
          {
            "class": "okhttp3.internal.connection.RealCall$AsyncCall",
            "method": "run",
            "file": "RealCall.kt",
            "line": 517,
            "exact": false,
            "location": "flink-kubernetes-operator-1.5.0-shaded.jar",
            "version": "1.5.0"
          }
        ]
      },
      "extendedStackTrace": [
        {
          "class": "io.fabric8.kubernetes.client.KubernetesClientException",
          "method": "launderThrowable",
          "file": "KubernetesClientException.java",
          "line": 129,
          "exact": false,
          "location": "flink-kubernetes-operator-1.5.0-shaded.jar",
          "version": "1.5.0"
        },
        {
          "class": "io.fabric8.kubernetes.client.KubernetesClientException",
          "method": "launderThrowable",
          "file": "KubernetesClientException.java",
          "line": 122,
          "exact": false,
          "location": "flink-kubernetes-operator-1.5.0-shaded.jar",
          "version": "1.5.0"
        },
        {
          "class": "io.fabric8.kubernetes.client.dsl.internal.BaseOperation",
          "method": "lambda$submitWatch$7",
          "file": "BaseOperation.java",
          "line": 665,
          "exact": false,
          "location": "flink-kubernetes-operator-1.5.0-shaded.jar",
          "version": "1.5.0"
        }
      ]
    },
    "extendedStackTrace": [
      {
        "class": "java.util.concurrent.CompletableFuture",
        "method": "encodeThrowable",
        "line": -1,
        "exact": false,
        "location": "?",
        "version": "?"
      },
      {
        "class": "java.util.concurrent.CompletableFuture",
        "method": "completeThrowable",
        "line": -1,
        "exact": false,
        "location": "?",
        "version": "?"
      },
      {
        "class": "java.util.concurrent.CompletableFuture",
        "method": "uniHandle",
        "line": -1,
        "exact": false,
        "location": "?",
        "version": "?"
      },
      {
        "class": "java.util.concurrent.CompletableFuture$UniHandle",
        "method": "tryFire",
        "line": -1,
        "exact": false,
        "location": "?",
        "version": "?"
      },
      {
        "class": "java.util.concurrent.CompletableFuture",
        "method": "postComplete",
        "line": -1,
        "exact": false,
        "location": "?",
        "version": "?"
      },
      {
        "class": "java.util.concurrent.CompletableFuture",
        "method": "completeExceptionally",
        "line": -1,
        "exact": false,
        "location": "?",
        "version": "?"
      },
      {
        "class": "io.fabric8.kubernetes.client.http.StandardHttpClient",
        "method": "lambda$buildWebSocket$13",
        "file": "StandardHttpClient.java",
        "line": 196,
        "exact": false,
        "location": "flink-kubernetes-operator-1.5.0-shaded.jar",
        "version": "1.5.0"
      },
      {
        "class": "java.util.concurrent.CompletableFuture",
        "method": "uniWhenComplete",
        "line": -1,
        "exact": false,
        "location": "?",
        "version": "?"
      },
      {
        "class": "java.util.concurrent.CompletableFuture$UniWhenComplete",
        "method": "tryFire",
        "line": -1,
        "exact": false,
        "location": "?",
        "version": "?"
      },
      {
        "class": "java.util.concurrent.CompletableFuture",
        "method": "postComplete",
        "line": -1,
        "exact": false,
        "location": "?",
        "version": "?"
      },
      {
        "class": "java.util.concurrent.CompletableFuture",
        "method": "completeExceptionally",
        "line": -1,
        "exact": false,
        "location": "?",
        "version": "?"
      },
      {
        "class": "io.fabric8.kubernetes.client.http.StandardHttpClient",
        "method": "lambda$completeOrCancel$5",
        "file": "StandardHttpClient.java",
        "line": 118,
        "exact": false,
        "location": "flink-kubernetes-operator-1.5.0-shaded.jar",
        "version": "1.5.0"
      },
      {
        "class": "io.fabric8.kubernetes.client.http.StandardHttpClient",
        "method": "lambda$retryWithExponentialBackoff$7",
        "file": "StandardHttpClient.java",
        "line": 159,
        "exact": false,
        "location": "flink-kubernetes-operator-1.5.0-shaded.jar",
        "version": "1.5.0"
      },
      {
        "class": "java.util.concurrent.CompletableFuture",
        "method": "uniWhenComplete",
        "line": -1,
        "exact": true,
        "location": "?",
        "version": "?"
      },
      {
        "class": "java.util.concurrent.CompletableFuture$UniWhenComplete",
        "method": "tryFire",
        "line": -1,
        "exact": true,
        "location": "?",
        "version": "?"
      },
      {
        "class": "java.util.concurrent.CompletableFuture",
        "method": "postComplete",
        "line": -1,
        "exact": true,
        "location": "?",
        "version": "?"
      },
      {
        "class": "java.util.concurrent.CompletableFuture",
        "method": "completeExceptionally",
        "line": -1,
        "exact": true,
        "location": "?",
        "version": "?"
      },
      {
        "class": "io.fabric8.kubernetes.client.okhttp.OkHttpWebSocketImpl$1",
        "method": "onFailure",
        "file": "OkHttpWebSocketImpl.java",
        "line": 90,
        "exact": true,
        "location": "flink-kubernetes-operator-1.5.0-shaded.jar",
        "version": "1.5.0"
      },
      {
        "class": "okhttp3.internal.ws.RealWebSocket",
        "method": "failWebSocket",
        "file": "RealWebSocket.kt",
        "line": 592,
        "exact": true,
        "location": "flink-kubernetes-operator-1.5.0-shaded.jar",
        "version": "1.5.0"
      },
      {
        "class": "okhttp3.internal.ws.RealWebSocket$connect$1",
        "method": "onFailure",
        "file": "RealWebSocket.kt",
        "line": 202,
        "exact": true,
        "location": "flink-kubernetes-operator-1.5.0-shaded.jar",
        "version": "1.5.0"
      },
      {
        "class": "okhttp3.internal.connection.RealCall$AsyncCall",
        "method": "run",
        "file": "RealCall.kt",
        "line": 525,
        "exact": true,
        "location": "flink-kubernetes-operator-1.5.0-shaded.jar",
        "version": "1.5.0"
      },
      {
        "class": "java.util.concurrent.ThreadPoolExecutor",
        "method": "runWorker",
        "line": -1,
        "exact": true,
        "location": "?",
        "version": "?"
      },
      {
        "class": "java.util.concurrent.ThreadPoolExecutor$Worker",
        "method": "run",
        "line": -1,
        "exact": true,
        "location": "?",
        "version": "?"
      },
      {
        "class": "java.lang.Thread",
        "method": "run",
        "line": -1,
        "exact": true,
        "location": "?",
        "version": "?"
      }
    ]
  },
  "endOfBatch": false,
  "loggerFqcn": "org.apache.logging.slf4j.Log4jLogger",
  "threadId": 38,
  "threadPriority": 5
}
To add more information which might be useful, I have set:
Copy code
kubernetes.operator.startup.stop-on-informer-error: false
and then deployed canaries as described in the docs: • https://nightlies.apache.org/flink/flink-kubernetes-operator-docs-release-1.6/docs/operations/health/ and everything seems to work fine...
Copy code
{"instant":{"epochSecond":1694094739,"nanoOfSecond":713000000},"thread":"pool-4-thread-2","level":"INFO","loggerName":"org.apache.flink.kubernetes.operator.health.CanaryResourceManager","message":"Canary deployment healthy","endOfBatch":false,"loggerFqcn":"org.apache.logging.slf4j.Log4jLogger","threadId":60,"threadPriority":5}
{"instant":{"epochSecond":1694094739,"nanoOfSecond":713000000},"thread":"pool-4-thread-2","level":"INFO","loggerName":"org.apache.flink.kubernetes.operator.health.CanaryResourceManager","message":"Scheduling canary check for ResourceID{name='canary', namespace='platform'} in 60s","endOfBatch":false,"loggerFqcn":"org.apache.logging.slf4j.Log4jLogger","threadId":60,"threadPriority":5}
{"instant":{"epochSecond":1694094739,"nanoOfSecond":727000000},"thread":"ReconcilerExecutor-flinkdeploymentcontroller-124","level":"INFO","loggerName":"org.apache.flink.kubernetes.operator.health.CanaryResourceManager","message":"Reconciling canary resource","endOfBatch":false,"loggerFqcn":"org.apache.logging.slf4j.Log4jLogger","threadId":124,"threadPriority":5}
{"instant":{"epochSecond":1694094739,"nanoOfSecond":933000000},"thread":"pool-4-thread-1","level":"INFO","loggerName":"org.apache.flink.kubernetes.operator.health.CanaryResourceManager","message":"Canary deployment healthy","endOfBatch":false,"loggerFqcn":"org.apache.logging.slf4j.Log4jLogger","threadId":59,"threadPriority":5}
{"instant":{"epochSecond":1694094739,"nanoOfSecond":934000000},"thread":"pool-4-thread-1","level":"INFO","loggerName":"org.apache.flink.kubernetes.operator.health.CanaryResourceManager","message":"Scheduling canary check for ResourceID{name='canary', namespace='platform-jobs'} in 60s","endOfBatch":false,"loggerFqcn":"org.apache.logging.slf4j.Log4jLogger","threadId":59,"threadPriority":5}
{"instant":{"epochSecond":1694094740,"nanoOfSecond":0},"thread":"ReconcilerExecutor-flinkdeploymentcontroller-125","level":"INFO","loggerName":"org.apache.flink.kubernetes.operator.health.CanaryResourceManager","message":"Reconciling canary resource","endOfBatch":false,"loggerFqcn":"org.apache.logging.slf4j.Log4jLogger","threadId":125,"threadPriority":5}
{"instant":{"epochSecond":1694094799,"nanoOfSecond":725000000},"thread":"pool-4-thread-3","level":"INFO","loggerName":"org.apache.flink.kubernetes.operator.health.CanaryResourceManager","message":"Canary deployment healthy","endOfBatch":false,"loggerFqcn":"org.apache.logging.slf4j.Log4jLogger","threadId":72,"threadPriority":5}
{"instant":{"epochSecond":1694094799,"nanoOfSecond":725000000},"thread":"pool-4-thread-3","level":"INFO","loggerName":"org.apache.flink.kubernetes.operator.health.CanaryResourceManager","message":"Scheduling canary check for ResourceID{name='canary', namespace='platform'} in 60s","endOfBatch":false,"loggerFqcn":"org.apache.logging.slf4j.Log4jLogger","threadId":72,"threadPriority":5}
{"instant":{"epochSecond":1694094799,"nanoOfSecond":798000000},"thread":"ReconcilerExecutor-flinkdeploymentcontroller-126","level":"INFO","loggerName":"org.apache.flink.kubernetes.operator.health.CanaryResourceManager","message":"Reconciling canary resource","endOfBatch":false,"loggerFqcn":"org.apache.logging.slf4j.Log4jLogger","threadId":126,"threadPriority":5}
{"instant":{"epochSecond":1694094799,"nanoOfSecond":999000000},"thread":"pool-4-thread-2","level":"INFO","loggerName":"org.apache.flink.kubernetes.operator.health.CanaryResourceManager","message":"Canary deployment healthy","endOfBatch":false,"loggerFqcn":"org.apache.logging.slf4j.Log4jLogger","threadId":60,"threadPriority":5}
{"instant":{"epochSecond":1694094799,"nanoOfSecond":999000000},"thread":"pool-4-thread-2","level":"INFO","loggerName":"org.apache.flink.kubernetes.operator.health.CanaryResourceManager","message":"Scheduling canary check for ResourceID{name='canary', namespace='platform-jobs'} in 60s","endOfBatch":false,"loggerFqcn":"org.apache.logging.slf4j.Log4jLogger","threadId":60,"threadPriority":5}
{"instant":{"epochSecond":1694094800,"nanoOfSecond":13000000},"thread":"ReconcilerExecutor-flinkdeploymentcontroller-128","level":"INFO","loggerName":"org.apache.flink.kubernetes.operator.health.CanaryResourceManager","message":"Reconciling canary resource","endOfBatch":false,"loggerFqcn":"org.apache.logging.slf4j.Log4jLogger","threadId":128,"threadPriority":5}
{"instant":{"epochSecond":1694094859,"nanoOfSecond":739000000},"thread":"pool-4-thread-1","level":"INFO","loggerName":"org.apache.flink.kubernetes.operator.health.CanaryResourceManager","message":"Canary deployment healthy","endOfBatch":false,"loggerFqcn":"org.apache.logging.slf4j.Log4jLogger","threadId":59,"threadPriority":5}
{"instant":{"epochSecond":1694094859,"nanoOfSecond":740000000},"thread":"pool-4-thread-1","level":"INFO","loggerName":"org.apache.flink.kubernetes.operator.health.CanaryResourceManager","message":"Scheduling canary check for ResourceID{name='canary', namespace='platform'} in 60s","endOfBatch":false,"loggerFqcn":"org.apache.logging.slf4j.Log4jLogger","threadId":59,"threadPriority":5}
{"instant":{"epochSecond":1694094859,"nanoOfSecond":755000000},"thread":"ReconcilerExecutor-flinkdeploymentcontroller-129","level":"INFO","loggerName":"org.apache.flink.kubernetes.operator.health.CanaryResourceManager","message":"Reconciling canary resource","endOfBatch":false,"loggerFqcn":"org.apache.logging.slf4j.Log4jLogger","threadId":129,"threadPriority":5}
{"instant":{"epochSecond":1694094860,"nanoOfSecond":11000000},"thread":"pool-4-thread-3","level":"INFO","loggerName":"org.apache.flink.kubernetes.operator.health.CanaryResourceManager","message":"Canary deployment healthy","endOfBatch":false,"loggerFqcn":"org.apache.logging.slf4j.Log4jLogger","threadId":72,"threadPriority":5}
{"instant":{"epochSecond":1694094860,"nanoOfSecond":12000000},"thread":"pool-4-thread-3","level":"INFO","loggerName":"org.apache.flink.kubernetes.operator.health.CanaryResourceManager","message":"Scheduling canary check for ResourceID{name='canary', namespace='platform-jobs'} in 60s","endOfBatch":false,"loggerFqcn":"org.apache.logging.slf4j.Log4jLogger","threadId":72,"threadPriority":5}
{"instant":{"epochSecond":1694094860,"nanoOfSecond":29000000},"thread":"ReconcilerExecutor-flinkdeploymentcontroller-130","level":"INFO","loggerName":"org.apache.flink.kubernetes.operator.health.CanaryResourceManager","message":"Reconciling canary resource","endOfBatch":false,"loggerFqcn":"org.apache.logging.slf4j.Log4jLogger","threadId":130,"threadPriority":5}
^C
Copy code
kgp -n flink-operator 
NAME                                         READY   STATUS    RESTARTS       AGE
flink-kubernetes-operator-686b889b45-x2dl9   1/1     Running   31 (30m ago)   5h10m
Shouldn't the canaries produce the same failure as before setting 
kubernetes.operator.startup.stop-on-informer-error
to 
false
?
9 Views
Open in Slack
PreviousNext
Powered by