Hi Team, I am continuously facing this issue. Can ...
# troubleshooting
m
Hi Team, I am continuously facing this issue. Can some from here help? I see this ticket is open too long and many user are facing too. https://issues.apache.org/jira/browse/FLINK-32212
Copy code
cdp-customer-profile-flink-hudi-taskmanager-1-3 @ ip-10-230-15-69.ap-south-1.compute.internal (dataPort=38719).
java.lang.IllegalStateException: The library registration references a different set of library BLOBs than previous registrations for this job:
old:[p-de4b1680d065109d59ef7958483755d33ccd99e2-2edc04a09cff45f9714e59581b6bb2dd]
new:[p-de4b1680d065109d59ef7958483755d33ccd99e2-bb81a92443d3bd2a99dc8fcfaad94962]
	at org.apache.flink.runtime.execution.librarycache.BlobLibraryCacheManager$ResolvedClassLoader.verifyClassLoader(BlobLibraryCacheManager.java:437)
	at org.apache.flink.runtime.execution.librarycache.BlobLibraryCacheManager$ResolvedClassLoader.access$600(BlobLibraryCacheManager.java:373)
	at org.apache.flink.runtime.execution.librarycache.BlobLibraryCacheManager$LibraryCacheEntry.getOrResolveClassLoader(BlobLibraryCacheManager.java:249)
	at org.apache.flink.runtime.execution.librarycache.BlobLibraryCacheManager$LibraryCacheEntry.access$1200(BlobLibraryCacheManager.java:210)
	at org.apache.flink.runtime.execution.librarycache.BlobLibraryCacheManager$DefaultClassLoaderLease.getOrResolveClassLoader(BlobLibraryCacheManager.java:350)
	at org.apache.flink.runtime.taskmanager.Task.createUserCodeClassloader(Task.java:1059)
	at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:637)
	at org.apache.flink.runtime.taskmanager.Task.run(Task.java:575)
	at java.base/java.lang.Thread.run(Unknown Source)
2025-08-11T05:54:03.092Z [flink-pekko.actor.default-dispatcher-13] INFO  o.a.f.r.s.adaptive.AdaptiveScheduler - Restarting job.
java.lang.IllegalStateException: The library registration references a different set of library BLOBs than previous registrations for this job:
old:[p-de4b1680d065109d59ef7958483755d33ccd99e2-2edc04a09cff45f9714e59581b6bb2dd]
new:[p-de4b1680d065109d59ef7958483755d33ccd99e2-bb81a92443d3bd2a99dc8fcfaad94962]
	at org.apache.flink.runtime.execution.librarycache.BlobLibraryCacheManager$ResolvedClassLoader.verifyClassLoader(BlobLibraryCacheManager.java:437)
	at org.apache.flink.runtime.execution.librarycache.BlobLibraryCacheManager$ResolvedClassLoader.access$600(BlobLibraryCacheManager.java:373)
	at org.apache.flink.runtime.execution.librarycache.BlobLibraryCacheManager$LibraryCacheEntry.getOrResolveClassLoader(BlobLibraryCacheManager.java:249)
	at org.apache.flink.runtime.execution.librarycache.BlobLibraryCacheManager$LibraryCacheEntry.access$1200(BlobLibraryCacheManager.java:210)
	at org.apache.flink.runtime.execution.librarycache.BlobLibraryCacheManager$DefaultClassLoaderLease.getOrResolveClassLoader(BlobLibraryCacheManager.java:350)
	at org.apache.flink.runtime.taskmanager.Task.createUserCodeClassloader(Task.java:1059)
	at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:637)
	at org.apache.flink.runtime.taskmanager.Task.run(Task.java:575)
	at java.base/java.lang.Thread.run(Unknown Source)
and seeing the task manager is not releaseing like mentioned on above ticket
Copy code
pod/customer-flink-hudi-6c654cddbd-bkx6l   1/1     Running   17 (62m ago)    18h
pod/customer-flink-hudi-taskmanager-17-1   1/1     Running       0               118m
pod/customer-flink-hudi-taskmanager-17-2   1/1     Running       0               118m
pod/customer-flink-hudi-taskmanager-17-3   1/1     Running       0               118m
pod/customer-flink-hudi-taskmanager-17-4   1/1     Running       0               118m
pod/customer-flink-hudi-taskmanager-17-5   1/1     Running       0               118m
pod/customer-flink-hudi-taskmanager-17-6   1/1     Running       0               118m
pod/customer-flink-hudi-taskmanager-18-1   1/1     Running       0               61m
pod/customer-flink-hudi-taskmanager-18-2   1/1     Running       0               61m
pod/customer-flink-hudi-taskmanager-18-3   1/1     Running       0               61m
pod/customer-flink-hudi-taskmanager-18-4   1/1     Running       0               61m
pod/customer-flink-hudi-taskmanager-18-5   1/1     Running       0               61m
pod/customer-flink-hudi-taskmanager-18-6   1/1     Running       0               61m
Hi Team, Can someone from the team take a look? Since upgrading Flink to version 1.20, it seems that the TaskManager is not being released properly when autoscaling is enabled. it is creating alot of issue.
m
What are the exact steps to reproduce this issue, we are also using flink autoscaling feature and recently we upgraded from flink1.16 to flink 1.20 and using kubernetes for HA. so may be this is relevant to our use case as well
m
@Mukul Gupta, it seems to occur randomly, most often during autoscaling events. During these times, the cache isn't cleared properly, and the older TaskManagers aren't being terminated as expected.