Lon Ilesanmi
07/05/2024, 2:01 PMcom.esotericsoftware.kryo.KryoException: Unable to find class: com.redacted.redacted.domain.dynamo.Customer
It happens very randomly and sometimes even when there is only configuration change, and no code change whatsoever. So far, we do not have a way to deterministically reproduce it. In our case, we do not explicitly use Kryo Serialization in any part of our codebase, so it must be Flink doing this under the hood.
I found this old thread with a similar exception log, but it seems the cause of that cases was already resolved as far back as version 1.0.0 based on this , while in our case we are using version release-1.16.0
Any idea what could be the cause of this?D. Draco O'Brien
07/06/2024, 9:13 AMD. Draco O'Brien
07/06/2024, 9:15 AMD. Draco O'Brien
07/06/2024, 9:17 AMD. Draco O'Brien
07/06/2024, 9:18 AMD. Draco O'Brien
07/06/2024, 9:18 AMLon Ilesanmi
07/10/2024, 9:41 AM#1 Ensure that the class is available in all flink job manager and task manager instances.Sorry if it sounds like a dumb question, but how do I confirm this?
# 2 if your app does dynamic classNo, we do not have any dynamic or custom class loading
# 3 check state.backend.serialization.typeWe do not override this setting anywhere
#4 check if there is a way to register the class with Kyro.Thanks, I will check this, but do you think it's really a case of Kryo registration failing, versus a case of the file being missing due to networking or some thing else during job/task manager start up - considering that this happens only when we do a redeployment
D. Draco O'Brien
07/10/2024, 9:46 AMLon Ilesanmi
07/10/2024, 9:52 AMLon Ilesanmi
07/10/2024, 10:02 AMKushtrim Morina
07/10/2024, 10:14 AMLon Ilesanmi
07/15/2024, 9:32 AMD. Draco O'Brien
07/15/2024, 9:37 AMD. Draco O'Brien
07/15/2024, 9:38 AMKushtrim Morina
07/15/2024, 9:55 AMD. Draco O'Brien
07/15/2024, 9:56 AMKushtrim Morina
07/15/2024, 9:57 AMD. Draco O'Brien
07/15/2024, 9:58 AMkubectl exec --stdin --tty shell-demo -- /bin/bash
this allows you to enter the shell of a running podD. Draco O'Brien
07/15/2024, 10:00 AMD. Draco O'Brien
07/15/2024, 10:00 AMLon Ilesanmi
07/15/2024, 10:02 AMD. Draco O'Brien
07/15/2024, 10:03 AMD. Draco O'Brien
07/15/2024, 10:04 AMD. Draco O'Brien
07/15/2024, 10:05 AMD. Draco O'Brien
07/15/2024, 10:05 AMD. Draco O'Brien
07/15/2024, 10:06 AMD. Draco O'Brien
07/15/2024, 10:06 AMD. Draco O'Brien
07/15/2024, 10:07 AMLon Ilesanmi
07/15/2024, 10:07 AMHave you isolated which container images and build is used when this error occurs?That's the thing, we deployed the same Docker image to multiple jobs and regions, and only some of them had the issue, only 4 out of about 45 jobs. The rest were fine
D. Draco O'Brien
07/15/2024, 10:08 AMLon Ilesanmi
07/15/2024, 10:09 AMD. Draco O'Brien
07/15/2024, 10:36 AMLon Ilesanmi
07/16/2024, 2:22 PMD. Draco O'Brien
07/17/2024, 7:27 AMD. Draco O'Brien
07/17/2024, 7:30 AMD. Draco O'Brien
07/17/2024, 7:32 AMLon Ilesanmi
07/17/2024, 9:41 AM