Terrence Tian
06/13/2024, 2:59 AMyux
06/13/2024, 3:07 AMTerrence Tian
06/13/2024, 3:14 AMTerrence Tian
06/13/2024, 3:16 AMTerrence Tian
06/13/2024, 3:21 AM# Local Schema
$ ./bin/flink run-application \
--target kubernetes-application \
-Dkubernetes.cluster-id=my-first-application-cluster \
-Dkubernetes.container.image.ref=custom-image-name \
local:///opt/flink/usrlib/my-flink-job.jar
I don't find the answer to my question in the documentation, so I need your help!Terrence Tian
06/13/2024, 3:21 AMyux
06/13/2024, 5:58 AM./bin/flink run-application \
--target kubernetes-application \
-Dkubernetes.cluster-id=my-first-application-cluster \
-Dkubernetes.container.image.ref=<IMAGE NAME HERE!> \
local:///opt/flink/usrlib/my-flink-job.jar
with in Flink Home locally and job should be successfully submitted to Kubernetes cluster.Terrence Tian
06/13/2024, 6:18 AM$ ./bin/flink run-application
Is it on my local computer?
My idea is to start the task by building an image containing the task, deploying it to the k8s cluster, and then running it immediately.
So how does the above command work? What does it do for me behind the scenes?Terrence Tian
06/13/2024, 6:20 AMyux
06/14/2024, 2:57 AMbash
minikube start
kubectl auth can-i create pods # yes
and authorize Flink to exeecute:
bash
create clusterrolebinding flink-role-binding-default --clusterrole=edit --serviceaccount=default:default
2. Enter minikube docker environment with eval $(minikube docker-env), write a Dockerfile as follows:
dockerfile
FROM flink:1.19.0-scala_2.12-java17
RUN mkdir -p $FLINK_HOME/usrlib
COPY WordCount.jar $FLINK_HOME/usrlib/my-flink-job.jar
and build it:
bash
docker build -t flink-appmode-test .
3. Run this on a machine where Kubectl points to created cluster (local machine here):
bash
./bin/flink run-application \
--target kubernetes-application \
-Dkubernetes.cluster-id=dummy-cluster \
-Dkubernetes.container.image.ref=flink-appmode-test \
local:///opt/flink/usrlib/my-flink-job.jar
4. Check if the job is running by listing pods:
bash
$kubectl get pods
NAME READY STATUS RESTARTS AGE
dummy-cluster-bf7467674-b7lws 1/1 Running 0 3s
and read its logs:
bash
$kubectl logs dummy-cluster-bf7467674-b7lws
sed: can't read /config.yaml: No such file or directory
/opt/flink/bin/config-parser-utils.sh: line 45: /opt/flink/conf/config.yaml: Read-only file system
Starting kubernetes-application as a console application on host dummy-cluster-bf7467674-b7lws.
WARNING: Unknown module: jdk.compiler specified to --add-exports
WARNING: Unknown module: jdk.compiler specified to --add-exports
WARNING: Unknown module: jdk.compiler specified to --add-exports
WARNING: Unknown module: jdk.compiler specified to --add-exports
WARNING: Unknown module: jdk.compiler specified to --add-exports
2024-06-14 02:56:16,049 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - --------------------------------------------------------------------------------
2024-06-14 02:56:16,050 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - Preconfiguration:
2024-06-14 02:56:16,050 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] -
...
Hopefully it helps :)Terrence Tian
06/14/2024, 3:15 AM