After I created my own image according to the offi...
# troubleshooting
t
After I created my own image according to the official documentation, I found an error when starting the container.
y
Hi Terrence, Could you please provide more information about the error, like container log outputs?
t
Sorry, I was busy and didn't submit a complete question.
According to the document, when you package an image yourself, you need to log in to the container and then execute the tasks you developed. This does not conform to the actual situation. I hope that the tasks will also start when the container starts. When I tried to do this, the above-mentioned problem occurred. https://nightlies.apache.org/flink/flink-docs-release-1.19/zh/docs/deployment/resource-providers/native_kubernetes/#application-mode
I don’t understand why I have to choose one of (jobmanager|standalone-job|taskmanager|history-server) since I want to use the native_kubernetes application-mode deployment mode.
Copy code
# Local Schema
$ ./bin/flink run-application \
    --target kubernetes-application \
    -Dkubernetes.cluster-id=my-first-application-cluster \
    -Dkubernetes.container.image.ref=custom-image-name \
    local:///opt/flink/usrlib/my-flink-job.jar
I don't find the answer to my question in the documentation, so I need your help!
I used image: flink:1.19.0-scala_2.12-java17
y
IIUC you don't need to actually start the packaged docker container locally nor execute anything inside. After built and tagged docker images, just run
Copy code
./bin/flink run-application \
    --target kubernetes-application \
    -Dkubernetes.cluster-id=my-first-application-cluster \
    -Dkubernetes.container.image.ref=<IMAGE NAME HERE!> \
    local:///opt/flink/usrlib/my-flink-job.jar
with in Flink Home locally and job should be successfully submitted to Kubernetes cluster.
t
Where is the following command executed?
Copy code
$ ./bin/flink run-application
Is it on my local computer? My idea is to start the task by building an image containing the task, deploying it to the k8s cluster, and then running it immediately. So how does the above command work? What does it do for me behind the scenes?
As I can run the task in IDEA by executing the main() method, I know that deploying a standalone app on k8s is just like deploying a normal Java app, but I don't see this on the documentation, so I'm confused.
y
Hi Terrence, here's my step-by-step attempts: 1. Start a MiniKube cluster locally with
Copy code
bash
minikube start
kubectl auth can-i create pods # yes
and authorize Flink to exeecute:
Copy code
bash
create clusterrolebinding flink-role-binding-default --clusterrole=edit --serviceaccount=default:default
2. Enter minikube docker environment with
eval $(minikube docker-env)
, write a Dockerfile as follows:
Copy code
dockerfile
FROM flink:1.19.0-scala_2.12-java17
RUN mkdir -p $FLINK_HOME/usrlib
COPY WordCount.jar $FLINK_HOME/usrlib/my-flink-job.jar
and build it:
Copy code
bash
docker build -t flink-appmode-test .
3. Run this on a machine where Kubectl points to created cluster (local machine here):
Copy code
bash
./bin/flink run-application \
    --target kubernetes-application \
    -Dkubernetes.cluster-id=dummy-cluster \
    -Dkubernetes.container.image.ref=flink-appmode-test \
    local:///opt/flink/usrlib/my-flink-job.jar
4. Check if the job is running by listing pods:
Copy code
bash
$kubectl get pods
NAME                            READY   STATUS    RESTARTS   AGE
dummy-cluster-bf7467674-b7lws   1/1     Running   0          3s
and read its logs:
Copy code
bash
$kubectl logs dummy-cluster-bf7467674-b7lws
sed: can't read /config.yaml: No such file or directory
/opt/flink/bin/config-parser-utils.sh: line 45: /opt/flink/conf/config.yaml: Read-only file system
Starting kubernetes-application as a console application on host dummy-cluster-bf7467674-b7lws.
WARNING: Unknown module: jdk.compiler specified to --add-exports
WARNING: Unknown module: jdk.compiler specified to --add-exports
WARNING: Unknown module: jdk.compiler specified to --add-exports
WARNING: Unknown module: jdk.compiler specified to --add-exports
WARNING: Unknown module: jdk.compiler specified to --add-exports
2024-06-14 02:56:16,049 INFO  org.apache.flink.runtime.entrypoint.ClusterEntrypoint        [] - --------------------------------------------------------------------------------
2024-06-14 02:56:16,050 INFO  org.apache.flink.runtime.entrypoint.ClusterEntrypoint        [] -  Preconfiguration:
2024-06-14 02:56:16,050 INFO  org.apache.flink.runtime.entrypoint.ClusterEntrypoint        [] -

...
Hopefully it helps :)
t
😘Thank you for taking the time to answer my question! I can understand the above example, but my question is how to deploy flink job in a declarative way.(This is the design idea of ​​​​k8s) Because I can't connect directly to the production k8s cluster, I can't publish directly through commands. Since my company uses GitOPS workflow, services must be published in a declarative way. The above imperative method is not suitable for my work environment. The solution I think of now is to describe resources through jsonnet, just like deploying ordinary Java APP, and finally ArgoCD helps me interact with k8s. The problem with this is that the official Flink image is useless to me, and I have to build the image myself.πŸ˜‚