Hi all, I'm trying to run <basic-checkpoint-ha.yam...
# troubleshooting
j
Hi all, I'm trying to run basic-checkpoint-ha.yaml with flink-kubernetes-operator. I installed minikube with all default settings, and deployed the operator according to quick start guide. However, I'm keep getting the following error:
Copy code
ERROR org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - Fatal error occurred in the cluster entrypoint.
java.util.concurrent.CompletionException: java.lang.IllegalStateException: The base directory of the JobResultStore isn't accessible. No dirty JobResults can be restored.
at java.util.concurrent.CompletableFuture.encodeThrowable(Unknown Source) ~[?:?]
at java.util.concurrent.CompletableFuture.completeThrowable(Unknown Source) [?:?]
at java.util.concurrent.CompletableFuture$AsyncSupply.run(Unknown Source) [?:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) [?:?]
at java.lang.Thread.run(Unknown Source) [?:?]
Caused by: java.lang.IllegalStateException: The base directory of the JobResultStore isn't accessible. No dirty JobResults can be restored.
at org.apache.flink.util.Preconditions.checkState(Preconditions.java:193) ~[flink-dist-1.16.1.jar:1.16.1]
at org.apache.flink.runtime.highavailability.FileSystemJobResultStore.getDirtyResultsInternal(FileSystemJobResultStore.java:182) ~[flink-dist-1.16.1.jar:1.16.1]
at org.apache.flink.runtime.highavailability.AbstractThreadsafeJobResultStore.withReadLock(AbstractThreadsafeJobResultStore.java:118) ~[flink-dist-1.16.1.jar:1.16.1]
at org.apache.flink.runtime.highavailability.AbstractThreadsafeJobResultStore.getDirtyResults(AbstractThreadsafeJobResultStore.java:100) ~[flink-dist-1.16.1.jar:1.16.1]
at org.apache.flink.runtime.dispatcher.runner.SessionDispatcherLeaderProcess.getDirtyJobResults(SessionDispatcherLeaderProcess.java:194) ~[flink-dist-1.16.1.jar:1.16.1]
at org.apache.flink.runtime.dispatcher.runner.AbstractDispatcherLeaderProcess.supplyUnsynchronizedIfRunning(AbstractDispatcherLeaderProcess.java:198) ~[flink-dist-1.16.1.jar:1.16.1]
at org.apache.flink.runtime.dispatcher.runner.SessionDispatcherLeaderProcess.getDirtyJobResultsIfRunning(SessionDispatcherLeaderProcess.java:188) ~[flink-dist-1.16.1.jar:1.16.1]
... 4 more
From previous discussions I learned this might be related to permission issues, so I tried the following fix:
Copy code
securityContext:
        runAsUser: 9999
        runAsGroup: 9999
        fsGroup: 9999
under
podTemplate
. But it's still not working and getting the same error. I'm wondering if I'm missing any configurations, or if anything not default should be applied to minikube. Any help is appreciated and thanks in advance. I'm using minikube v1.25.2 and Flink v1.16.1, if it matters.
✅ 1
Mark myself as solved. The problem is caused by the
hostPath
used in the configuration. According to K8S, it's only writable by root. Running the application as root won't work, because it involves further adjusting the configuration files. The easy fix is to find the path in minikube container, and set its permission to 777 (
chmod -R 777
). Now the problem can be solved.