Hi
Is there some workaround for this issue:
https://issues.apache.org/jira/browse/FLINK-32212 ?
GCP cluster was upgrading node pools over my night and for half hour task manager was down with this exception in a loop.
~3.58 AM: First job manager was taken down, replaced by second. Task manager reconnected to job manager successfully.
~4.10 AM: For some reason, I see another cancellation and reschedule, but task manager and job manager pods were same
~4.13 AM: Second job manager was taken down, by another node upgrade by GCP cluster. Task manager cancelled job and has fallen into infinite loop of accepting job and having exception when loading JAR.
~4.50 AM: Node pool for task managers got upgraded, and restarted pod loaded well without issues. Env has recovered, but there was more than 30 minutes of downtime in the result.
Seems deleting task manager resolved this issue.
I feel like it may become a be more widespread issue?
Using flink kubernetes operator (Starting Flink Kubernetes Operator (Version: 1.9.0, Flink Version: 1.18.1, Rev:17129ff, Date
2024 06 18T0743
22+0200)), flink job is using 1_19 version (I wrote it just before 1_20 appeared). However, considering the ticket I found, upgrading to 1_20 will probably not change much.
Fix well... seems restarting task manager. Easy, but I wonder if there is a way to detect & restart without manual intervention. Or flink could perhaps just drop all the cache in case of errors like this? If there is a mismatch, then new distribution would probably be needed?
Thanks!