Flink Job : SSL handshake failed Caused by: javax.net.ssl.SSLHandshakeException: PKIX path validation failed: java.security.cert.CertPathValidatorException: Path does not chain with any of the trust anchors
Flink version : 1.20.1
Environment : Hybrid system - onprem running with K3s cluster and aws cloud ( eks , msk , etc..)
K3s cluster (SUSE)
Flink Kubernetes operator to manage Flink deployments
Strimzi Kafka
We have k3s cluster running with strimiz kafka , flink operator.
We have deployed Flink Jobs that read data from strimzi kafka and do a remote write/produce to AWS cloud MSK cluster.
The Flink jobs have secured communication with Strimzi Kafka with self signed certificates .
We have Flink jobs with AOS and checkpointing enabled and jobs without checkpointing.
Happy Scenario : The Flink jobs are consuming from Strimzi Kafka and successfully producing to AWS Cloud MSK
Failure Scenario : The certificates have expiry date of 1 year before certificates expires it needs to be rotated
When the certificates are rotated the Flink Job Manager is restarting to pick the new secrets(certificates are stored as kubernetes secrets) . On restarting the job manager contains the SSL handshake errors and the job in the taskmanager pod is restarting continuously.
These SSL errors are observed in jobs that have checkpointing .
Fix that was tried.
1. kubectl delete po of jobmanager and taskmanager(4 attempts) - error still exists
2. kubectl edit deployment - update env variable - new jobmanager started with same SSL error - issue exists
3. state clean up - rm -rf checkpoints/ , rm -rf ha/ , scaled the deployment to 0 , removed the cluster-config-map and job-config-map , scaled the deployment to 1
New jobmanager and taskmanager pod are running and the job also in running state - no errors , SSL issue resolved.
Recovering the job after this error requires manual intervention which is not something one would like to do in a production system.
Why the Flink Job is behaving like this and after clean up of the state and configmap , how is it recovering. How do we fix this issue.
Note : The certificates are valid as it is working for Flink jobs which are not having checkpointing enabled.
If the errors continue and after fixed restart attempts the job is going to failed state and when we restart the pod we job
id not found error.
There are other issues we are faceing in our Flink job , probably will create a separate thread for each issue . This being critical one
Please find the diagram if its helps in understanding the issue better.
Please find the logs of jobmanager and taskmanager.