Hello, I am trying Flink-Kubernetes-Operator on AK...
# troubleshootingz
Zhiyu Tian
03/15/2023, 4:28 AMHello, I am trying Flink-Kubernetes-Operator on AKS cluster and met an error when submitting an example job https://github.com/apache/flink-kubernetes-operator/blob/main/examples/basic.yaml.
Could you help on this?
The error message from operator:
2023-03-15 042025,969 o.a.f.c.d.a.c.ApplicationClusterDeployer [INFO ] [magnetar.basic-example] Submitting application in 'Application Mode'.
2023-03-15 042026,010 o.a.f.k.u.KubernetesUtils [INFO ] [magnetar.basic-example] Kubernetes deployment requires a fixed port. Configuration blob.server.port will be set to 6124
2023-03-15 042026,010 o.a.f.k.u.KubernetesUtils [INFO ] [magnetar.basic-example] Kubernetes deployment requires a fixed port. Configuration taskmanager.rpc.port will be set to 6122
2023-03-15 042026,011 o.a.f.c.Configuration [INFO ] [magnetar.basic-example] Config uses fallback configuration key 'kubernetes.service-account' instead of key 'kubernetes.jobmanager.service-account'
2023-03-15 042026,030 o.a.f.k.KubernetesClusterDescriptor [WARN ] [magnetar.basic-example] Failed to create the Kubernetes cluster "basic-example", try to clean up the residual resources.
2023-03-15 042026,048 o.a.f.k.o.l.AuditUtils [INFO ] [magnetar.basic-example] >>> Event | Warning | CLUSTERDEPLOYMENTEXCEPTION | Could not create Kubernetes cluster "basic-example".
2023-03-15 042026,048 o.a.f.k.o.r.ReconciliationUtils [WARN ] [magnetar.basic-example] Attempt count: 0, last attempt: false
2023-03-15 042026,084 o.a.f.k.o.l.AuditUtils [INFO ] [magnetar.basic-example] >>> Status | Error | UPGRADING | {"type":"org.apache.flink.kubernetes.operator.exception.ReconciliationException","message":"org.apache.flink.client.deployment.ClusterDeploymentException: Could not create Kubernetes cluster \"basic-example\".","throwableList":[{"type":"org.apache.flink.client.deployment.ClusterDeploymentException","message":"Could not create Kubernetes cluster \"basic-example\"."},{"type":"org.apache.flink.kubernetes.shaded.io.fabric8.kubernetes.client.KubernetesClientException","message":"Failure executing: POST at: https://10.0.0.1/apis/apps/v1/namespaces/magnetar/deployments. Message: Internal error occurred: failed calling webhook \"deploymentwebhook.microsoft.com\": failed to call webhook: yaml: mapping values are not allowed in this context. Received status: Status(apiVersion=v1, code=500, details=StatusDetails(causes=[StatusCause(field=null, message=failed calling webhook \"deploymentwebhook.microsoft.com\": failed to call webhook: yaml: mapping values are not allowed in this context, reason=null, additionalProperties={})], group=null, kind=null, name=null, retryAfterSeconds=null, uid=null, additionalProperties={}), kind=Status, message=Internal error occurred: failed calling webhook \"deploymentwebhook.microsoft.com\": failed to call webhook: yaml: mapping values are not allowed in this context, metadata=ListMeta(_continue=null, remainingItemCount=null, resourceVersion=null, selfLink=null, additionalProperties={}), reason=InternalError, status=Failure, additionalProperties={})."}]}
2023-03-15 042026,084 i.j.o.p.e.ReconciliationDispatcher [ERROR] [magnetar.basic-example] Error during event processing ExecutionScope{ resource id: ResourceID{name='basic-example', namespace='magnetar'}, version: 124702780} failed.
org.apache.flink.kubernetes.operator.exception.ReconciliationException: org.apache.flink.client.deployment.ClusterDeploymentException: Could not create Kubernetes cluster "basic-example".
at org.apache.flink.kubernetes.operator.controller.FlinkDeploymentController.reconcile(FlinkDeploymentController.java:142)
u
01 ico
03/15/2023, 4:30 AMmay be you should go to k8s community for this question
r
Ricco Førgaard
03/15/2023, 7:29 AMTry disabling the webhook. Put this in your
values.yaml Copy code
webhook:
create: falsez
Zhiyu Tian
03/15/2023, 10:44 AMThank you for the advice.
I disable the flink-kubernetes-operator webhook, and re-submit the job, it failed again.
The error message showed that the validation work comes from the Deployment webhook of the AKS cluster.
Zhiyu Tian
03/16/2023, 8:11 AM1. Change the log level from info to debug, and got the debug message which is attached.
2. Extract the JobManager Deployment from the error message.
3. changing the resource format
4. Run "kubectl apply -f ", and it failed with unknown fields: .additionalProperties, finalizers, etc....
5. Run "kubectl apply -f xx.yaml --validate=false", it succeeded.
6. So, I think/guess there may be some additional checks in the webhook of AKS.
s
Sumit Nekar
03/16/2023, 4:15 PMHave u installed certificate manager ?
https://nightlies.apache.org/flink/flink-kubernetes-operator-docs-main/docs/try-flink-kubernetes-operator/quick-start/
z
Zhiyu Tian
03/17/2023, 1:51 AMI have disabled the webhook for the Flink operator. But it still did not work.
Zhiyu Tian
03/18/2023, 6:07 AMFound the root cause of it.
A webhook has been deployed by another team to monitor Deployments in our Kubernetes cluster. This webhook has a functionality to check the namespace of a Deployment, and throw an exception if it determines that the namespace is null.
However, there seems to be a bug in the webhook's logic that causes it to throw exceptions even when the Deployment object is created with a null namespace by the namespaced Kubernetes client in Flink.
Another team is currently working on fixing the bug in the webhook.
đź‘Ť 1
2 Views