Flink Job : java.lang.IllegalStateException: The l...
# troubleshooting
j
Flink Job : java.lang.IllegalStateException: The library registration references a different set of library BLOBs than previous registrations for this job Hi Team we are getting the above error in the jobmanager pod and the job is not able to recover . Given : The jobmanager pod and taskmananger pod is running and processing data When : Jobmanager pod goes down and comes up Then : We observe the above error the job is not recovering Flink version used is 1.20 and no checkpointing enabled
Copy code
java.lang.IllegalStateException: The library registration references a different set of library BLOBs than previous registrations for this job:
old:[p-66da0997da3ef6aab1e1b2f3a39642e1af795d86-4b1cbc8c77187a1616ef9c739ada3dfa]
new:[p-66da0997da3ef6aab1e1b2f3a39642e1af795d86-c0819ab19fbbfff87951011e6a535764]
	at org.apache.flink.runtime.execution.librarycache.BlobLibraryCacheManager$ResolvedClassLoader.verifyClassLoader(BlobLibraryCacheManager.java:437) ~[flink-dist-1.20.1.jar:1.20.1]
	at org.apache.flink.runtime.execution.librarycache.BlobLibraryCacheManager$ResolvedClassLoader.access$600(BlobLibraryCacheManager.java:373) ~[flink-dist-1.20.1.jar:1.20.1]
	at org.apache.flink.runtime.execution.librarycache.BlobLibraryCacheManager$LibraryCacheEntry.getOrResolveClassLoader(BlobLibraryCacheManager.java:249) ~[flink-dist-1.20.1.jar:1.20.1]
	at org.apache.flink.runtime.execution.librarycache.BlobLibraryCacheManager$LibraryCacheEntry.access$1200(BlobLibraryCacheManager.java:210) ~[flink-dist-1.20.1.jar:1.20.1]
	at org.apache.flink.runtime.execution.librarycache.BlobLibraryCacheManager$DefaultClassLoaderLease.getOrResolveClassLoader(BlobLibraryCacheManager.java:350) ~[flink-dist-1.20.1.jar:1.20.1]
	at org.apache.flink.runtime.taskmanager.Task.createUserCodeClassloader(Task.java:1059) ~[flink-dist-1.20.1.jar:1.20.1]
	at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:637) ~[flink-dist-1.20.1.jar:1.20.1]
	at org.apache.flink.runtime.taskmanager.Task.run(Task.java:575) ~[flink-dist-1.20.1.jar:1.20.1]
	at java.lang.Thread.run(Unknown Source) ~[?:?]
m
Hi @Jaiprasad, Have you found any solution for this issue? I'm experiencing the same problem, particularly with autoscaling jobs, and it's occurring quite frequently. If you've already resolved it, could you please share your solution?
j
Hi @Mahesh Gupta - at present this error can be resolved by implementing checkpointing. We have a usecase where we do not need checkpointing and encountering the BLOB issues.
m
@Jaiprasad I already have checkpointing but still facing the same issue.
j
Oh ok so far we have not encountered this error with Flink workloads that have checkpoint enabled. 1. Check the configmap that get created for respective flink workload that contains the hash of the library 2. In the pvc or volume where the blob library is stored , check on that as well