We have been trying to use <pluggable file system>...
# troubleshooting
s
We have been trying to use pluggable file system to integrate our custom file system implementation in flink. However, we have been facing multiple Class not found issue due to this limitation mentioned in flink docs - "_In practice, it means you should avoid using
Thread.currentThread().getContextClassLoader()
class loader in your implementation_." In this case, our dependencies like jersey client and hadoop are using thread's class loader to initialise some classes which we have no control over. Has anyone faced this issue ?? Isn't this limitation too restrictive because it would be very hard to reimplement your dependencies ??
m
Those classes that you have no control offer, are you implying that they are outside of your file system implementation? Plugins cannot access classes from either other plugins or from Flink itself, as outlined on https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/filesystems/plugins/
s
They are part of my plugin implementation. For ex - I am using jersey client to make a rest call to persist bytes on a remote object store. Whenever any dependency tries to create java object like
Class.forName("ClassName")
, It fails with Class Not found exception because
T_hread.currentThread().getContextClassLoader()
will return FlinkUserClassLoader instead PluginClassLoader._
I don't want my plugins to use classes from other plugins and flink. All the dependencies are already part of my plugin jar in a separate plugin jar folder. But when they use thread's context class loader, it returns the _FlinkUserClassLoader b_ecause parent thread in a call hierarchy belongs to flink-dist.jar.
m
If they are in a separate plugin JAR folder, then they are not part of your plugin. You should have all dependencies in one specific plugin folder