Hi team, Seeing JobManager failure to restart wit...
# troubleshooting
k
Hi team, Seeing JobManager failure to restart with the following error, has anyone seen this before?
Copy code
20 tasks will be restarted to recover the failed task 10fa6513f35966426dfd36576bff2bae_2878b9ab6776d58d3ac8f74bba6fe037_0_3879.
at java.lang.Thread.run(Unknown Source) ~[?:?]
at org.apache.flink.runtime.taskmanager.Task.run(Task.java:550) ~[flink-dist-1.16.1.jar:1.16.1]
at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:612) ~[flink-dist-1.16.1.jar:1.16.1]
at org.apache.flink.runtime.taskmanager.Task.createUserCodeClassloader(Task.java:1024) ~[flink-dist-1.16.1.jar:1.16.1]
at org.apache.flink.runtime.execution.librarycache.BlobLibraryCacheManager$DefaultClassLoaderLease.getOrResolveClassLoader(BlobLibraryCacheManager.java:336) ~[flink-dist-1.16.1.jar:1.16.1]
at org.apache.flink.runtime.execution.librarycache.BlobLibraryCacheManager$LibraryCacheEntry.access$1100(BlobLibraryCacheManager.java:202) ~[flink-dist-1.16.1.jar:1.16.1]
at org.apache.flink.runtime.execution.librarycache.BlobLibraryCacheManager$LibraryCacheEntry.getOrResolveClassLoader(BlobLibraryCacheManager.java:235) ~[flink-dist-1.16.1.jar:1.16.1]
at org.apache.flink.runtime.execution.librarycache.BlobLibraryCacheManager$ResolvedClassLoader.access$500(BlobLibraryCacheManager.java:359) ~[flink-dist-1.16.1.jar:1.16.1]
at org.apache.flink.runtime.execution.librarycache.BlobLibraryCacheManager$ResolvedClassLoader.verifyClassLoader(BlobLibraryCacheManager.java:419) ~[flink-dist-1.16.1.jar:1.16.1]
new:[p-143a36af2df48da99d11f33fe47fff0418ca5511-ba29eb7b159adeb0e4cba1e5b379399f]
old:[p-143a36af2df48da99d11f33fe47fff0418ca5511-f37b157bf869ea7e830900ee42ceb4dd]
java.lang.IllegalStateException: The library registration references a different set of library BLOBs than previous registrations for this job:
2023-07-12 15:05:22,500 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Source: Kinesis Data Stream -> watermarking_strategy_v0 -> Flat Map (1/16) (10fa6513f35966426dfd36576bff2bae_2878b9ab6776d58d3ac8f74bba6fe037_0_3879) switched from DEPLOYING to FAILED on eventlogger-pipeline-v0-taskmanager-18-1 @ ip-10-1-10-12.us-east-2.compute.internal (dataPort=42471).
Seeing the hash
old:[p-143a36af2df48da99d11f33fe47fff0418ca5511-f37b157bf869ea7e830900ee42ceb4dd]
that is being referenced related to KinesisDataFetcher, but not sure where the new one would be coming from. Any ideas?
cluster is running with 1 job manager but otherwise still in high availability mode