Hi, I'm battling to crack the problem of running the S3 Filesystem with the ORC format. Orc local works fine. JSON s3 works fine. Using them together seems challenging. Current setup: hadoop shaded in /lib orc format in /lib s3-fs-hadoop s3 in plugins/s3-fs-hadoop

HADOOP_OPTS="-Dfs.AbstractFileSystem.s3a.impl=org.apache.hadoop.fs.s3a.S3A -Dfs.s3a.impl=org.apache.hadoop.fs.s3a.S3AFileSystem"

I get an

Class org.apache.hadoop.fs.s3a.S3AFileSystem not found.

I've tried the plugins in lib, and that worked locally, but not on the cluster. Does anyone have an example setup?

Have you configured your Flink setup to include the Hadoop dependencies? https://nightlies.apache.org/flink/flink-docs-release-1.16/docs/dev/configuration/advanced/#hadoop-dependencies

I've added the flink-shaded-hadoop-2-uber-2.8.3-10.0.jar, as I understand it does the same thing?

I haven’t seen that being used for the last couple of years to be honest. From my experience is that you need to properly have Hadoop configured if you want to use a Hadoop file format

OK - that gives me something to go on. Thanks much, and I'll likely write a quick 'this is how' if I get it up and running

Good to hear you’ve got it to work