Bao Pham
02/14/2025, 12:47 AMjava.lang.IllegalStateException: The library registration references a different set of library BLOBs than previous registrations for this job:
old:[p-f062ec11b4fb8e3cc51e3e5c3a30ac76b6adbcf2-ea2ffce27e0b570602fd10fac32dd3ae]
new:[p-f062ec11b4fb8e3cc51e3e5c3a30ac76b6adbcf2-f779bd34646c61c1a0b6e5237cb63861]
at org.apache.flink.runtime.execution.librarycache.BlobLibraryCacheManager$ResolvedClassLoader.verifyClassLoader(BlobLibraryCacheManager.java:437)
at org.apache.flink.runtime.execution.librarycache.BlobLibraryCacheManager$ResolvedClassLoader.access$600(BlobLibraryCacheManager.java:373)
at org.apache.flink.runtime.execution.librarycache.BlobLibraryCacheManager$LibraryCacheEntry.getOrResolveClassLoader(BlobLibraryCacheManager.java:249)
at org.apache.flink.runtime.execution.librarycache.BlobLibraryCacheManager$LibraryCacheEntry.access$1200(BlobLibraryCacheManager.java:210)
at org.apache.flink.runtime.execution.librarycache.BlobLibraryCacheManager$DefaultClassLoaderLease.getOrResolveClassLoader(BlobLibraryCacheManager.java:350)
at org.apache.flink.runtime.taskmanager.Task.createUserCodeClassloader(Task.java:1042)
at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:624)
at org.apache.flink.runtime.taskmanager.Task.run(Task.java:562)
at java.base/java.lang.Thread.run(Unknown Source)
we are using Flink version 1.18.1. Can you help how to solve this issue? This makes our apps unstable and crash daily.Mahesh Gupta
02/14/2025, 7:28 AMMahesh Gupta
02/15/2025, 4:59 PMThe library registration references a different set of library BLOBs than previous registrations for this job:
old:[p-063eaba518e29e94eaaf7767b34add064972a93f-8f57ae1a4a89d1b25dfb9dcff1d8cb79]
new:[p-063eaba518e29e94eaaf7767b34add064972a93f-7e091b89d4e47df1cdafa57da1ad925f]
at org.apache.flink.runtime.execution.librarycache.BlobLibraryCacheManager$ResolvedClassLoader.verifyClassLoader(BlobLibraryCacheManager.java:437) ~[flink-dist-1.18.1.jar:1.18.1]
at org.apache.flink.runtime.execution.librarycache.BlobLibraryCacheManager$ResolvedClassLoader.access$600(BlobLibraryCacheManager.java:373) ~[flink-dist-1.18.1.jar:1.18.1]
at org.apache.flink.runtime.execution.librarycache.BlobLibraryCacheManager$LibraryCacheEntry.getOrResolveClassLoader(BlobLibraryCacheManager.java:249) ~[flink-dist-1.18.1.jar:1.18.1]
at org.apache.flink.runtime.execution.librarycache.BlobLibraryCacheManager$LibraryCacheEntry.access$1200(BlobLibraryCacheManager.java:210) ~[flink-dist-1.18.1.jar:1.18.1]
at org.apache.flink.runtime.execution.librarycache.BlobLibraryCacheManager$DefaultClassLoaderLease.getOrResolveClassLoader(BlobLibraryCacheManager.java:350) ~[flink-dist-1.18.1.jar:1.18.1]
at org.apache.flink.runtime.taskmanager.Task.createUserCodeClassloader(Task.java:1042) ~[flink-dist-1.18.1.jar:1.18.1]
at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:624) ~[flink-dist-1.18.1.jar:1.18.1]
at org.apache.flink.runtime.taskmanager.Task.run(Task.java:562) ~[flink-dist-1.18.1.jar:1.18.1]
at java.lang.Thread.run(Unknown Source) ~[?:?]
2025-02-15 16:56:45,117 INFO org.apache.flink.runtime.scheduler.adaptive.AdaptiveScheduler [] - Restarting job.
java.lang.IllegalStateException: The library registration references a different set of library BLOBs than previous registrations for this job:
old:[p-063eaba518e29e94eaaf7767b34add064972a93f-8f57ae1a4a89d1b25dfb9dcff1d8cb79]
new:[p-063eaba518e29e94eaaf7767b34add064972a93f-7e091b89d4e47df1cdafa57da1ad925f]
at org.apache.flink.runtime.execution.librarycache.BlobLibraryCacheManager$ResolvedClassLoader.verifyClassLoader(BlobLibraryCacheManager.java:437) ~[flink-dist-1.18.1.jar:1.18.1]
at org.apache.flink.runtime.execution.librarycache.BlobLibraryCacheManager$ResolvedClassLoader.access$600(BlobLibraryCacheManager.java:373) ~[flink-dist-1.18.1.jar:1.18.1]
at org.apache.flink.runtime.execution.librarycache.BlobLibraryCacheManager$LibraryCacheEntry.getOrResolveClassLoader(BlobLibraryCacheManager.java:249) ~[flink-dist-1.18.1.jar:1.18.1]
at org.apache.flink.runtime.execution.librarycache.BlobLibraryCacheManager$LibraryCacheEntry.access$1200(BlobLibraryCacheManager.java:210) ~[flink-dist-1.18.1.jar:1.18.1]
at org.apache.flink.runtime.execution.librarycache.BlobLibraryCacheManager$DefaultClassLoaderLease.getOrResolveClassLoader(BlobLibraryCacheManager.java:350) ~[flink-dist-1.18.1.jar:1.18.1]
at org.apache.flink.runtime.taskmanager.Task.createUserCodeClassloader(Task.java:1042) ~[flink-dist-1.18.1.jar:1.18.1]
at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:624) ~[flink-dist-1.18.1.jar:1.18.1]
at org.apache.flink.runtime.taskmanager.Task.run(Task.java:562) ~[flink-dist-1.18.1.jar:1.18.1]
at java.lang.Thread.run(Unknown Source) ~[?:?]Mahesh Gupta
02/19/2025, 5:20 AMThomas Cooper
02/19/2025, 11:35 AMMahesh Gupta
02/19/2025, 12:05 PMJasmin Redzepovic
02/19/2025, 12:12 PMMahesh Gupta
02/19/2025, 12:23 PMThomas Cooper
02/19/2025, 1:02 PMThomas Cooper
02/19/2025, 1:02 PMThomas Cooper
02/19/2025, 1:03 PMMahesh Gupta
02/19/2025, 1:39 PMThomas Cooper
02/19/2025, 1:41 PMThomas Cooper
02/19/2025, 1:42 PMMahesh Gupta
02/19/2025, 1:46 PMThomas Cooper
02/19/2025, 1:46 PMrandom in it several times (which is less than encouraging).
I would have to dig into it more to see if that actually does imply a different hashcode for every instance of BlobKey, even for the same items.Thomas Cooper
02/19/2025, 1:47 PMThomas Cooper
02/19/2025, 1:48 PMMahesh Gupta
02/19/2025, 1:49 PMThomas Cooper
02/19/2025, 2:48 PMBao Pham
03/24/2025, 5:07 PMBao Pham
03/24/2025, 5:08 PMBao Pham
03/24/2025, 5:09 PMThomas Cooper
03/24/2025, 5:13 PMThomas Cooper
03/24/2025, 5:14 PMBao Pham
03/24/2025, 5:24 PMBao Pham
03/24/2025, 5:26 PMThomas Cooper
03/24/2025, 5:26 PMThomas Cooper
03/24/2025, 5:27 PMBao Pham
03/24/2025, 5:28 PMFLINK_JAR_URI configured to point to our jarThomas Cooper
03/24/2025, 5:30 PMBao Pham
03/24/2025, 5:30 PMflinkVersion: v1_18
image: artifactory.com/docker/images/app:v1
job:
jarURI: local:///opt/flink/usrlib/classpath/app.jarBao Pham
03/24/2025, 5:33 PMDownloading 86158719a75f92da39ac66eeb87749c4/p-2a6f241c42591d974fbad067faa3a314d7aff8b3-1a1c1d35cb6655fe2efad285fc947dc5 from app/host:port
Downloading 86158719a75f92da39ac66eeb87749c4/p-2a6f241c42591d974fbad067faa3a314d7aff8b3-762be47ae0db0652d51334f2fa050b21 from app/host:port
And this matches with error
The library registration references a different set of library BLOBs than previous registrations for this job:
old:[p-2a6f241c42591d974fbad067faa3a314d7aff8b3-762be47ae0db0652d51334f2fa050b21]
new:[p-2a6f241c42591d974fbad067faa3a314d7aff8b3-1a1c1d35cb6655fe2efad285fc947dc5]
and I don't know what those messages meanBao Pham
03/24/2025, 5:50 PMBao Pham
03/31/2025, 4:21 PM