https://flink.apache.org/ logo
Join Slack
Powered by
Hi all, I am currently working on adding a new GC...
# troubleshooting
t
Hi all, I am currently working on adding a new GCS FileSource Connector to a job. Generally I will run my job in my IntelliJ locally to test and debug. However this is the first time I'm including a FileSystem as a Source and I'm hitting the problem of the Plugin Jar. We've used the gs hadoop plugin for our Checkpoint/Savepoints for awhile however those are not on when running locally in the IDE. Does anyone have a pointer on how I can get the hadoop plugin in a place that the IDE run will recognize and load?
Copy code
Caused by: org.apache.flink.core.fs.UnsupportedFileSystemSchemeException: Could not find a file system implementation for scheme 'gs'. The scheme is directly supported by Flink through the following plugin(s): flink-gs-fs-hadoop. Please ensure that each plugin resides within its own subfolder within the plugins directory. See <https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/plugins/> for more information. If you want to use a Hadoop file system for that scheme, please add the scheme to the configuration fs.allowed-fallback-filesystems. For a full list of supported file systems, please see <https://nightlies.apache.org/flink/flink-docs-stable/ops/filesystems/>.
Just unsure where the Jar should go for local runs like this
For anyone else having the issue of getting your FileSystems to work when running from IntelliJ: I was doing this when setting up my Env
Copy code
StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();

    if (config.env() == EnvEnum.DEV) {
      env = StreamExecutionEnvironment.createLocalEnvironmentWithWebUI(new Configuration());
    }
So I was able to simply add the PluginUtils and include a 
plugins/gs-fs-hadoop/xxx.jar
alongside my 
main/java
so the updated conditional is
Copy code
StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();

    if (config.env() == EnvEnum.DEV) {
      env = StreamExecutionEnvironment.createLocalEnvironmentWithWebUI(new Configuration());
      var pluginManager = PluginUtils.createPluginManagerFromRootFolder(new Configuration());
      FileSystem.initialize(new Configuration(), pluginManager);
    }
This will at minimum load the Plugins at startup in the IDE.
Open in Slack
PreviousNext
Powered by