Hi, I need help. Our applications often crash due ...
# troubleshooting
b
Hi, I need help. Our applications often crash due to this error
Copy code
java.lang.IllegalStateException: The library registration references a different set of library BLOBs than previous registrations for this job:
old:[p-f062ec11b4fb8e3cc51e3e5c3a30ac76b6adbcf2-ea2ffce27e0b570602fd10fac32dd3ae]
new:[p-f062ec11b4fb8e3cc51e3e5c3a30ac76b6adbcf2-f779bd34646c61c1a0b6e5237cb63861]
	at org.apache.flink.runtime.execution.librarycache.BlobLibraryCacheManager$ResolvedClassLoader.verifyClassLoader(BlobLibraryCacheManager.java:437)
	at org.apache.flink.runtime.execution.librarycache.BlobLibraryCacheManager$ResolvedClassLoader.access$600(BlobLibraryCacheManager.java:373)
	at org.apache.flink.runtime.execution.librarycache.BlobLibraryCacheManager$LibraryCacheEntry.getOrResolveClassLoader(BlobLibraryCacheManager.java:249)
	at org.apache.flink.runtime.execution.librarycache.BlobLibraryCacheManager$LibraryCacheEntry.access$1200(BlobLibraryCacheManager.java:210)
	at org.apache.flink.runtime.execution.librarycache.BlobLibraryCacheManager$DefaultClassLoaderLease.getOrResolveClassLoader(BlobLibraryCacheManager.java:350)
	at org.apache.flink.runtime.taskmanager.Task.createUserCodeClassloader(Task.java:1042)
	at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:624)
	at org.apache.flink.runtime.taskmanager.Task.run(Task.java:562)
	at java.base/java.lang.Thread.run(Unknown Source)
we are using Flink version 1.18.1. Can you help how to solve this issue? This makes our apps unstable and crash daily.
m
@Bao Pham, I’ve encountered this issue as well, but it’s intermittent on my end. I'm posting here to stay updated in case a solution is found. As mentioned, it’s intermittent, but it’s hard to reproduce.
I got this error today aswell
Copy code
The library registration references a different set of library BLOBs than previous registrations for this job:
old:[p-063eaba518e29e94eaaf7767b34add064972a93f-8f57ae1a4a89d1b25dfb9dcff1d8cb79]
new:[p-063eaba518e29e94eaaf7767b34add064972a93f-7e091b89d4e47df1cdafa57da1ad925f]
	at org.apache.flink.runtime.execution.librarycache.BlobLibraryCacheManager$ResolvedClassLoader.verifyClassLoader(BlobLibraryCacheManager.java:437) ~[flink-dist-1.18.1.jar:1.18.1]
	at org.apache.flink.runtime.execution.librarycache.BlobLibraryCacheManager$ResolvedClassLoader.access$600(BlobLibraryCacheManager.java:373) ~[flink-dist-1.18.1.jar:1.18.1]
	at org.apache.flink.runtime.execution.librarycache.BlobLibraryCacheManager$LibraryCacheEntry.getOrResolveClassLoader(BlobLibraryCacheManager.java:249) ~[flink-dist-1.18.1.jar:1.18.1]
	at org.apache.flink.runtime.execution.librarycache.BlobLibraryCacheManager$LibraryCacheEntry.access$1200(BlobLibraryCacheManager.java:210) ~[flink-dist-1.18.1.jar:1.18.1]
	at org.apache.flink.runtime.execution.librarycache.BlobLibraryCacheManager$DefaultClassLoaderLease.getOrResolveClassLoader(BlobLibraryCacheManager.java:350) ~[flink-dist-1.18.1.jar:1.18.1]
	at org.apache.flink.runtime.taskmanager.Task.createUserCodeClassloader(Task.java:1042) ~[flink-dist-1.18.1.jar:1.18.1]
	at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:624) ~[flink-dist-1.18.1.jar:1.18.1]
	at org.apache.flink.runtime.taskmanager.Task.run(Task.java:562) ~[flink-dist-1.18.1.jar:1.18.1]
	at java.lang.Thread.run(Unknown Source) ~[?:?]
2025-02-15 16:56:45,117 INFO  org.apache.flink.runtime.scheduler.adaptive.AdaptiveScheduler [] - Restarting job.
java.lang.IllegalStateException: The library registration references a different set of library BLOBs than previous registrations for this job:
old:[p-063eaba518e29e94eaaf7767b34add064972a93f-8f57ae1a4a89d1b25dfb9dcff1d8cb79]
new:[p-063eaba518e29e94eaaf7767b34add064972a93f-7e091b89d4e47df1cdafa57da1ad925f]
	at org.apache.flink.runtime.execution.librarycache.BlobLibraryCacheManager$ResolvedClassLoader.verifyClassLoader(BlobLibraryCacheManager.java:437) ~[flink-dist-1.18.1.jar:1.18.1]
	at org.apache.flink.runtime.execution.librarycache.BlobLibraryCacheManager$ResolvedClassLoader.access$600(BlobLibraryCacheManager.java:373) ~[flink-dist-1.18.1.jar:1.18.1]
	at org.apache.flink.runtime.execution.librarycache.BlobLibraryCacheManager$LibraryCacheEntry.getOrResolveClassLoader(BlobLibraryCacheManager.java:249) ~[flink-dist-1.18.1.jar:1.18.1]
	at org.apache.flink.runtime.execution.librarycache.BlobLibraryCacheManager$LibraryCacheEntry.access$1200(BlobLibraryCacheManager.java:210) ~[flink-dist-1.18.1.jar:1.18.1]
	at org.apache.flink.runtime.execution.librarycache.BlobLibraryCacheManager$DefaultClassLoaderLease.getOrResolveClassLoader(BlobLibraryCacheManager.java:350) ~[flink-dist-1.18.1.jar:1.18.1]
	at org.apache.flink.runtime.taskmanager.Task.createUserCodeClassloader(Task.java:1042) ~[flink-dist-1.18.1.jar:1.18.1]
	at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:624) ~[flink-dist-1.18.1.jar:1.18.1]
	at org.apache.flink.runtime.taskmanager.Task.run(Task.java:562) ~[flink-dist-1.18.1.jar:1.18.1]
	at java.lang.Thread.run(Unknown Source) ~[?:?]
Hi Team, Could someone from the official team please take a look at this?
t
Those errors are triggered from the UserClassLoader. What is your setup? How are you deploying Flink? Cluster session mode, application mode? How are you exposing your custom code to Flink?
m
Hi @Thomas Cooper thank you for checking this out. we are using k8s cluster to deploy our application and using helm to maintain the life-cycle of application. simple use case is to write data into hudi table by doing some transformation.
j
m
Yes @Jasmin Redzepovic it seems it very old issue and we are yet to resolved it.
t
Ok that issue answers my next question, do you seen this after restarts/node drains?
If you do this could be a sequencing issue with restarts between the Job and Task managers
It might be possible to get the operator (or something separate) to intercept the node draining requests and do a rolling restart of the deployment
m
@Thomas Cooper After restarting the job, it begins functioning normally. However, it is a bit challenging to restart the job every time since the issue occurs intermittently.
t
Well the operator could handle the restart automatically, however there may be a way to prevent it being an issue in the first place. It is not clear to me why there should be an issue if nothing has changed in the job other than the JobManager restarting.
< thinking out loud > I wonder if there is a way to do more deterministic checks for changes in the blobs?
m
I believe this issue might be caused by a cached or dangling thread that fails to release the old blob address, or it could be referencing a garbage address that no longer exists. but not 100% sure
t
The validity checks that trigger that exception are based on the lengths of the current and intended array of libraries/classpaths and their contents matching exactly. Because the logging shows there is one item in the old and the new it is down to how they are compared. That seems to be based on hashcode of the BlobKey class, which worryingly has the word
random
in it several times (which is less than encouraging). I would have to dig into it more to see if that actually does imply a different hashcode for every instance of BlobKey, even for the same items.
We would need a reliable reproducer to be able to test this theory
From the issue it sounds like there isn't one, maybe more detailed logging would be useful
m
Yes, the issue is difficult to reproduce and occurs intermittently(at will 😄).
t
Ok, that BlobKey hashcode random was a false positive.
b
I can tell you how to reproduce the issue. In the application, you use flink version 1.18.1 version, but then the dependency library using version different from version 1.18.1, then deploy the application, it works fine. After it runs for a while, then K8s restart our pods, then it faces this issue, it can't restart our pods, our application crash and loops forever.
This is easiest way to reproduce. However, even when we use all Flink version 1.18.1, then we still have this issue occurs, and in this scenario, I don't know what is the root cause. cc @Kieren Dilworth
This makes our applications not reliable since we have to manually restart our pods whenever it happens, and it occurs very frequently, almost every day. @Lee Wallen
t
So you are using library code in your Flink jobs that uses a different base flink version than the one your are running on?
How are you building your job jars?
b
Our library core just uses flink-core 1.18.0, our application uses 1.18.1. The easiest issue we see is last time our application automatically update flink-connector-jdbc from 3.2.0-1.18 to 3.2.0-1.19, then our applications start crashing every day with this issue. We build jar using gradle Jib library.
We then revert back flink-connector-jdbc from 3.2.0-1.19 -> 3.2.0-1.18, then we don't see this issue but last week we did another deployment, all library using flink version 1.18.1 but still crashing.
t
So how are those jars deployed, they are built into an image I assume, how is that image deployed?
Are your Flink deployments referencing versioned images or do they just point to "latest"
b
We use jib to build it as image, upload to artifactory, then have
FLINK_JAR_URI
configured to point to our jar
t
I was wondering if there is a way your Flink deployment could end up with some task managers pointing to the old jars and some to the new?
b
For example, this is Flink deployment configuration
Copy code
flinkVersion: v1_18
    image: artifactory.com/docker/images/app:v1
    job:
      jarURI: local:///opt/flink/usrlib/classpath/app.jar
I don't know about it, but when checking logs when K8s restart, I see this one
Copy code
Downloading 86158719a75f92da39ac66eeb87749c4/p-2a6f241c42591d974fbad067faa3a314d7aff8b3-1a1c1d35cb6655fe2efad285fc947dc5 from app/host:port
Downloading 86158719a75f92da39ac66eeb87749c4/p-2a6f241c42591d974fbad067faa3a314d7aff8b3-762be47ae0db0652d51334f2fa050b21 from app/host:port
And this matches with error
Copy code
The library registration references a different set of library BLOBs than previous registrations for this job:
old:[p-2a6f241c42591d974fbad067faa3a314d7aff8b3-762be47ae0db0652d51334f2fa050b21]
new:[p-2a6f241c42591d974fbad067faa3a314d7aff8b3-1a1c1d35cb6655fe2efad285fc947dc5]
and I don't know what those messages mean
would you like me to share our dependencies output?
Hey, I added HA and restart strategy to our flink deployment and it seems to solve the issue. So far I haven't seen the issue when K8s restarts our pods. Thanks for your help on this!
👍 1