Hi all, we randomly face this issue when (re-)depl...
# troubleshooting
l
Hi all, we randomly face this issue when (re-)deploying some of our Flink jobs.
Copy code
com.esotericsoftware.kryo.KryoException: Unable to find class: com.redacted.redacted.domain.dynamo.Customer
It happens very randomly and sometimes even when there is only configuration change, and no code change whatsoever. So far, we do not have a way to deterministically reproduce it. In our case, we do not explicitly use Kryo Serialization in any part of our codebase, so it must be Flink doing this under the hood. I found this old thread with a similar exception log, but it seems the cause of that cases was already resolved as far back as version 1.0.0 based on this , while in our case we are using version release-1.16.0 Any idea what could be the cause of this?
d
Ok, so sometimes its not finding the class. #1 Ensure that the class is available in all flink job manager and task manager instances.
# 2 if your app does dynamic class loading make sure any classes required for deserialization are accessible via default classloading especially check classloader.resolve-order which determines order flinks loads classes.
# 3 check state.backend.serialization.type check if there might be something in the settings affecting the use of Kyro
#4 check if there is a way to register the class with Kyro.
are you seeing this in dev/testing or production? or on across more than one environment?
l
Thanks for the reply @D. Draco O'Brien! We see it in production but very randomly - say once in a couple of weeks, and we deploy pretty much everyday; sometimes multiple times a day
#1 Ensure that the class is available in all flink job manager and task manager instances.
Sorry if it sounds like a dumb question, but how do I confirm this?
# 2 if your app does dynamic class
No, we do not have any dynamic or custom class loading
# 3 check state.backend.serialization.type
We do not override this setting anywhere
#4 check if there is a way to register the class with Kyro.
Thanks, I will check this, but do you think it's really a case of Kryo registration failing, versus a case of the file being missing due to networking or some thing else during job/task manager start up - considering that this happens only when we do a redeployment
d
#1 class availability. How are you deploying flink? There are several ways and this can affect how this is checked and guaranteed.
l
We are deploying in K8s via ArgoCD
(Cc. @Aqib Mehmood @Kushtrim Morina who are also on my team)
k
We use helm charts, which are synced using ArgoCD. For every new version of jobs we build a new Container Image and we do the upgrade of the image in FlinkDeployment manifest. This triggers the rollout of task/job managers
✅ 1
l
Hi @D. Draco O'Brien, did Kushtrim answer your question?
d
Yes, I see. So it looks like you are deploying docker images? Is there any patterns in terms of the pod or machine on which this is deployed? Are you bare metal? or AWS/GCP/Azure?
What are you using for CNI plugin? This could affect the networking aspect.
k
Hey Draco, We are using EKS cluster (AWS). Nodes are r5n.2xlarge. For CNI, we use AWS-VPC-CNI
d
And there were no changes or upgrades to CNI around the time of the anomalies?
k
No, there are not any changes on the cni during the anomalies
d
ok
Copy code
kubectl exec --stdin --tty shell-demo -- /bin/bash
this allows you to enter the shell of a running pod
You can use this to check for differences is classpath or networking access. pings etc
Are there patterns related to the pod/region etc associated with this issue?
l
The last time it happened, it was with our highest traffic jobs in asia and london regions - so it seems to correlate a bit with throughput
d
“We use helm charts, which are synced using ArgoCD. For every new version of jobs we build a new Container Image and we do the upgrade of the image in FlinkDeployment manifest. This triggers the rollout of task/job managers”
Have you isolated which container images and build is used when this error occurs?
I am going to take a guess that some timeouts need to be bumped up related to networking/timeouts
and they are needed when you are under high throughput or the network is under heavy loads.
You should also search logs for any timeout errors on the pod
or retries etc
and this might have to be done as part of the deployment topology on AWS
l
Have you isolated which container images and build is used when this error occurs?
That's the thing, we deployed the same Docker image to multiple jobs and regions, and only some of them had the issue, only 4 out of about 45 jobs. The rest were fine
d
That kind of rules out how you are creating the container i think. Its most likely a networking issue. Is the pod running on the same type machine when this error occurs?
l
Yes, we use the same machine type for all our task and job manager pods
d
ok, what are the timeout settings?
l
Hi @D. Draco O'Brien, sorry for the late response. We do not override any of the Flink operator's default timeout settings. @Kushtrim Morina please cmiiw
d
Ok, you can look at maybe bumping up some of the timeouts. akka.ask.timeout, task.cancellation.interval, taskmanager.cancel.timeout are ones to look at.
also taskmanager.network.idle.timeout in this case healthy but temporarily idle connections might get incorrectly closed out causing tasks to fail. In a highly distributed environment or under certain load conditions this might be set too low and need to be increased.
taskmanager.network.fetcher.timeout.ms can also cause issues with fetch operations during shuffles to fail prematurely. This can happen under high load or high network latency.
l
Thank you Draco! We will check all of those options and adjust them. And If the issue happens again and we still cannot find the cause, I will try to gather more information and share here