Hello. how can I use S3 as a streaming file sink. ...
# troubleshootingr
Reme Ajayi
02/08/2023, 9:53 PMHello. how can I use S3 as a streaming file sink. I have put fs-s3-hadoop jar in my plugins folder and added access keys to
flink-conf.yaml. Copy code
Could not find a file system implementation for scheme 's3'. The scheme is directly supported by Flink through the following plugin(s): flink-s3-fs-hadoop, flink-s3-fs-presto. Please ensure that each plugin resides within its own subfolder within the plugins directory. See <https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/plugins/> for more information. If you want to use a Hadoop file system for that scheme, please add the scheme to the configuration fs.allowed-fallback-filesystems. For a full list of supported file systems, please see <https://nightlies.apache.org/flink/flink-docs-stable/ops/filesystems/>.m
Martijn Visser
02/09/2023, 7:36 AMHave you added the JAR in a subfolder in the plugins directory?
r
Reme Ajayi
02/09/2023, 1:05 PMYes I have
m
Martijn Visser
02/09/2023, 1:10 PMCan you check the logs when starting the cluster that this JAR is indeed loaded when starting?
r
Reme Ajayi
02/09/2023, 1:14 PMI checked the jobmanager and taskmanager logs from the Flink UI(please let me know if there is a preferred way to check this). This jar is not included in the jars that were added to my classpath. The only jars there are the ones included in my pom.xml
Reme Ajayi
02/09/2023, 1:16 PMI installed flink locally. The application with the code I'm running is a separate project, that I build as jar and run in the terminal through the flink directory.
Reme Ajayi
02/09/2023, 1:49 PMRestarted my cluster and it works, thanks!
g
Gaurav Miglani
02/10/2023, 3:15 PMalso try using s3a
h
Hangyu Wang
05/31/2023, 7:17 AMI also got the same error, and I have restarted the local cluster but still not working. Any other ideas? Many thanks.
Hangyu Wang
05/31/2023, 7:27 AMm
Martijn Visser
05/31/2023, 7:29 AMYour image shows a download folder for Flink 1.17 and the S3 JAR for 1.17.1.
h
Hangyu Wang
05/31/2023, 7:35 AMI changed the 1.17.1 version to 1.17.0 and restarted the cluster, still not working.
Hangyu Wang
05/31/2023, 7:37 AMHow could I check that the plugin has been successfully loaded?
Hangyu Wang
05/31/2023, 7:38 AMm
Martijn Visser
05/31/2023, 7:39 AMSo it's loaded π
h
Hangyu Wang
05/31/2023, 7:44 AM Copy code
Caused by: org.apache.flink.util.FlinkRuntimeException: Could not enumerate file splits
at org.apache.flink.connector.file.src.AbstractFileSource.createEnumerator(AbstractFileSource.java:143)
at org.apache.flink.runtime.source.coordinator.SourceCoordinator.start(SourceCoordinator.java:213)
... 8 more
Caused by: org.apache.flink.core.fs.UnsupportedFileSystemSchemeException: Could not find a file system implementation for scheme 's3'. The scheme is directly supported by Flink through the following plugin(s): flink-s3-fs-hadoop, flink-s3-fs-presto. Please ensure that each plugin resides within its own subfolder within the plugins directory. See <https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/plugins/> for more information. If you want to use a Hadoop file system for that scheme, please add the scheme to the configuration fs.allowed-fallback-filesystems. For a full list of supported file systems, please see <https://nightlies.apache.org/flink/flink-docs-stable/ops/filesystems/>.
at org.apache.flink.core.fs.FileSystem.getUnguardedFileSystem(FileSystem.java:515)
at org.apache.flink.core.fs.FileSystem.get(FileSystem.java:409)
at org.apache.flink.core.fs.Path.getFileSystem(Path.java:274)
at org.apache.flink.connector.file.src.enumerate.NonSplittingRecursiveEnumerator.enumerateSplits(NonSplittingRecursiveEnumerator.java:82)
at org.apache.flink.connector.file.src.AbstractFileSource.createEnumerator(AbstractFileSource.java:141)Hangyu Wang
05/31/2023, 7:44 AMThis is the error I have now.
9 Views