Hello community, I launched Cluster via Flink k8s operator, and when i try to launch the job via UI ( upload jar and submit from UI any flink examples ) it runs the job but when i login into the Job manager POD and try to run ./flink run examples/sample.jar. It throws following exception. Any idea what might be wrong here ?

The program finished with the following exception:

org.apache.flink.client.program.ProgramInvocationException: The main method caused an error: Failure executing: GET at: <https://10.43.0.1/api/v1/namespaces/apl-edm-streaming-svcs-l3-dev1/services/session-seven-deployment-only-example-rest>. Message: Forbidden!Configured service account doesn't have access. Service account may have been revoked. services "session-seven-deployment-only-example-rest" is forbidden: User "system:serviceaccount:apl-edm-streaming-svcs-l3-dev1:flink" cannot get resource "services" in API group "" in the namespace "apl-edm-streaming-svcs-l3-dev1".
	at org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:372)
	at org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:222)
	at org.apache.flink.client.ClientUtils.executeProgram(ClientUtils.java:105)
	at org.apache.flink.client.cli.CliFrontend.executeProgram(CliFrontend.java:851)
	at org.apache.flink.client.cli.CliFrontend.run(CliFrontend.java:245)
	at org.apache.flink.client.cli.CliFrontend.parseAndRun(CliFrontend.java:1095)
	at org.apache.flink.client.cli.CliFrontend.lambda$mainInternal$9(CliFrontend.java:1189)
	at org.apache.flink.runtime.security.contexts.NoOpSecurityContext.runSecured(NoOpSecurityContext.java:28)
	at org.apache.flink.client.cli.CliFrontend.mainInternal(CliFrontend.java:1189)
	at org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:1157)
Caused by: org.apache.flink.kubernetes.shaded.io.fabric8.kubernetes.client.KubernetesClientException: Failure executing: GET at: <https://10.43.0.1/api/v1/namespaces/apl-edm-streaming-svcs-l3-dev1/services/session-seven-deployment-only-example-rest>. Message: Forbidden!Configured service account doesn't have access. Service account may have been revoked. services "session-seven-deployment-only-example-rest" is forbidden: User "system:serviceaccount:apl-edm-streaming-svcs-l3-dev1:flink" cannot get resource "services" in API group "" in the namespace "apl-edm-streaming-svcs-l3-dev1".

Permissions kubectl get serviceaccounts NAME SECRETS AGE flink 1 154d flink-operator 1 154d kubectl get roles NAME CREATED AT flink 2023-02-08T185229Z flink-operator 2023-07-11T205124Z MU-C02CF4WYMD6R:seven rxahuja$ kubectl describe role flink Name: flink Labels: app.kubernetes.io/managed-by=Helm app.kubernetes.io/name=flink-kubernetes-operator app.kubernetes.io/version=1.5.0 helm.sh/chart=flink-kubernetes-operator-1.5.0 Annotations: helm.sh/resource-policy: keep meta.helm.sh/release-name: flink-kubernetes-operator meta.helm.sh/release-namespace: apl-edm-streaming-svcs-l3-dev1 PolicyRule: Resources Non-Resource URLs Resource Names Verbs --------- ----------------- -------------- ----- configmaps [] [] [*] pods [] [] [*] deployments.apps/finalizers [] [] [*] deployments.apps [] [] [*] MU-C02CF4WYMD6R:seven rxahuja$ kubectl describe role flink-operator Name: flink-operator Labels: app.kubernetes.io/managed-by=Helm app.kubernetes.io/name=flink-kubernetes-operator app.kubernetes.io/version=1.6-SNAPSHOT helm.sh/chart=flink-kubernetes-operator-1.6-SNAPSHOT Annotations: <none> PolicyRule: Resources Non-Resource URLs Resource Names Verbs --------- ----------------- -------------- ----- configmaps [] [] [*] deployments [] [] [*] events [] [] [*] flinkdeployments.flink.apache.org [] [] [*] pods [] [] [*] replicasets [] [] [*] services [] [] [*] deployments.apps/finalizers [] [] [*] deployments.apps [] [] [*]

As the error says, your service account doesn't have access to list services. You will need to create a k8s role and rolebinding and attach the rolebinding to your service account flink. Just lookup Google or ask chatgpt about it 😉

Thanks a lot, yes that worked.