Hey all, should I be able to run 3 instances of Jo...
# troubleshooting
r
Hey all, should I be able to run 3 instances of JobManager in HA, each in a different namespace (i.e. flink-dev, flink-qa, flink-stg) on a Kubernetes Cluster with Flink Kubernetes Operator 1.8. I made sure each of these namespaces have RBAC for Flink properly setup. It is having problem with the leader election and updating the cluster-configmap in each namespace. They are all generating the following error:
Copy code
Caused by: java.util.concurrent.CompletionException: org.apache.flink.runtime.persistence.PossibleInconsistentStateException: org.apache.flink.kubernetes.shaded.io.fabric8.kubernetes.client.KubernetesClientException: Failure executing: PUT at: https://172.1.0.1:443/api/v1/namespaces/flink-dev/configmaps/flink-v-119-cluster-config-map. Message: Operation cannot be fulfilled on configmaps "flink-v-119-cluster-config-map": the object has been modified; please apply your changes to the latest version and try again. Received status: Status(apiVersion=v1, code=409, details=StatusDetails(causes=[], group=null, kind=configmaps, name=flink-v-119-cluster-config-map, retryAfterSeconds=null, uid=null, additionalProperties={}), kind=Status, message=Operation cannot be fulfilled on configmaps "flink-v-119-cluster-config-map": the object has been modified; please apply your changes to the latest version and try again, metadata=ListMeta(_continue=null, remainingItemCount=null, resourceVersion=null, selfLink=null, additionalProperties={}), reason=Conflict, status=Failure, additionalProperties={}).
P.S My annotations for my flink deployment is as follows:
Copy code
high-availability.storageDir: 'file:///flink-data/ha'
high-availability.type: kubernetes
jobmanager.memory.jvm-metaspace.size: 1024m
state.checkpoints.dir: 'file:///flink-data/checkpoints'
state.savepoints.dir: 'file:///flink-data/savepoints'
taskmanager.numberOfTaskSlots: '20'
g
Do you get this already at startup? Even if you only run a single Flink app in a given namespace?
r
Yes, to be clear, I just deploying a FlinkDeployment resource with the annotation above. If I removed the HA annotation, the pod comes up and I am able to deploy FlinkSessionJobs that works, but I have no HA. For example, the JobManager restarted and I had to delete and restart all my session jobs to get them working again.
Here is the yaml that I tried to deploy....it works if I remove the high-availability annotations. With the high-availability annotations, it can elect the leader and says it can't update the cluster configmap that I assumed the operator is auto generating because there is already a change:
Copy code
apiVersion: flink.apache.org/v1beta1
kind: FlinkDeployment
metadata:
  name: flink-v-119
spec:
  image: flink:1.19
  flinkVersion: v1_19
  flinkConfiguration:
    taskmanager.numberOfTaskSlots: "20"
    state.savepoints.dir: file:///flink-data/savepoints
    state.checkpoints.dir: file:///flink-data/checkpoints
    high-availability.type: kubernetes
    high-availability.storageDir: file:///flink-data/ha
    jobmanager.memory.jvm-metaspace.size: 1024m
    env.java.opts.jobmanager:
      -Dkubernetes.max.concurrent.requests=1000
  jobManager:
    replicas: 2
    resource:
      memory: 2048m
      cpu: 1
  taskManager:
    resource:
      memory:  2048m
      cpu: 1
  podTemplate:
    spec:
      containers:
        - name: flink-main-container
          volumeMounts:
          - name: flink-vol
            mountPath: /flink-data
          - name: "kv-mongocloud-xxxxx"
            mountPath: "/var/secrets/kv-mongocloud-xxxxxx"
            readOnly: true
          - name: "kv-k8s-xxxxx"
            mountPath: "/var/secrets/kv-k8s-xxxxx"
            readOnly: true
          - name: "kv-kafka-xxxxxxx"
            mountPath: "/var/secrets/kv-kafka-xxxxxxxx"
            readOnly: true
          envFrom:
          - configMapRef:
              name: flink-configmap
          - secretRef:
              name: flink-mongodb-secrets
          - secretRef:
              name: flink-secrets
          - secretRef:
              name: flink-kafka-secrets
      volumes:
        - name: "kv-mongocloud-xxxxxxx"
          csi:
            driver: secrets-store.csi.k8s.io
            readOnly: true
            volumeAttributes:
              secretProviderClass: "release-name-kv-mongocloud-xxxxx"
        - name: "kv-k8s-stg-001-inline"
          csi:
            driver: secrets-store.csi.k8s.io
            readOnly: true
            volumeAttributes:
              secretProviderClass: "release-name-kv-k8s-xxxxx"
        - name: "kv-kafka-xxxxxxxxxxxxxxx"
          csi:
            driver: secrets-store.csi.k8s.io
            readOnly: true
            volumeAttributes:
              secretProviderClass: "release-name-kv-kafka-xxxxxxx"
        - name: flink-vol
          persistentVolumeClaim:
            claimName: flink-pvc-stg
  serviceAccount: flink
g
The HA config maps are generally not controlled by the operator, only Flink job itself
r
Interesting....are you talking about a FlinkSessionJob? If so, I can't imagine why, here is what my session jobs look like:
Copy code
---
# Source: flink-sql-jobs/templates/flink-session-job.yaml
apiVersion: flink.apache.org/v1beta1
kind: FlinkSessionJob
metadata:
  name: job-offer-accepted-forwarder
spec:
  deploymentName: flink-v-119
  restartNonce: 1700
  job:
    jarURI: https://artifactory.xxx.com:443/artifactory/xxx.maven.ado.dev.local/sql-runner-1.5.jar
    args: ["--fname=flink-lead-offer-accepted-forwarder-1.7.sql"]
    parallelism: 2
    upgradeMode: savepoint
    state: running
    allowNonRestoredState: true
---
# Source: flink-sql-jobs/templates/flink-session-job.yaml
apiVersion: flink.apache.org/v1beta1
kind: FlinkSessionJob
metadata:
  name: job-knative-to-sidekick-forwarder
spec:
  deploymentName: flink-v-119
  restartNonce: 1700
  job:
    jarURI: https://artifactory.xxx.com:443/artifactory/xxx.maven.ado.dev.local/sql-runner-1.5.jar
    args: ["--fname=knative-to-sidekick-forwarder-1.7.sql"]
    parallelism: 1
    upgradeMode: savepoint
    state: running
I have nothing else deployed, just the FlinkDeployment above and the FlinkSessionJobs that I also have an example above. Is there something one with either of those yaml declarations?
@Gyula Fóra Anyone I can possibly reach out that might have an insight. This has been killing me. BTW, it is running on Azure Kubernetes Service in case it makes any difference.
g
What happens if you deploy the included HA example? That seems to reliable work everywhere: https://github.com/apache/flink-kubernetes-operator/blob/main/examples/basic-checkpoint-ha.yaml
r
I was just about to write to you....I finally thought about it and figured out the issue had to be some kind of race condition. I googled and found some settings that I tried and it has been running now in HA for 9 hours. The settings are as follows but I am not sure if some of these values are too high:
Copy code
high-availability.kubernetes.leader-election.lease-duration: 300s
high-availability.kubernetes.leader-election.renew-deadline: 90s
high-availability.kubernetes.leader-election.retry-period: 15s
Thanks for you help, please let me know if should bring any of these values down. Lastly, I am now having an issue with pulling stuff from the web for example: https://artifactory.xxx.com:443/artifactory/xxx.maven.ado.dev.local/sql-runner-1.5.jar. Should that be working in HA?
@Gyula Fóra FYI, changing the storageDir to use the wasbs protocol fixed my issue. Thanks again for your help.
👍 1