Hey everyone I am struggling to follow the documentation for Apache Flink #troubleshooting

Hey everyone. I am struggling to follow the docum...

Amresh Venugopal

03/02/2023, 8:19 AM

Hey everyone. I am struggling to follow the documentation for using s3 plugins. This is how I have integrated the plugins as per the File Systems doc.

Copy code

❯ tree plugins
plugins
├── README.txt
└── s3-fs-presto
    └── flink-s3-fs-presto-1.16.1.jar

I'll add my flink-conf.yaml in the thread. When I run:

Copy code

flink run some.jar

I see the following exception in the

flink-amresh.venugopal-client-ip-172-31-50-66.log

. The

.jar

I am using works if I don't use s3 for checkpointing.

Copy code

Caused by: org.apache.flink.core.fs.UnsupportedFileSystemSchemeException: Could not find a file system implementation for scheme 's3p'. The scheme is directly supported by Flink through the following plugin(s): flink-s3-fs-presto. Please ensure that each plugin resides within its own subfolder within the plugins directory. See <https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/plugins/> for more information. If you want to use a Hadoop file system for that scheme, please add the scheme to the configuration fs.allowed-fallback-filesystems. For a full list of supported file systems, please see <https://nightlies.apache.org/flink/flink-docs-stable/ops/filesystems/>.
	at org.apache.flink.core.fs.FileSystem.getUnguardedFileSystem(FileSystem.java:515)
	at org.apache.flink.core.fs.FileSystem.get(FileSystem.java:409)
	at org.apache.flink.core.fs.Path.getFileSystem(Path.java:274)

Amresh Venugopal

03/02/2023, 8:21 AM

I have also followed this stackoverflow answer but results in the same error message. The following is my flink-conf

Copy code

jobmanager.rpc.address: localhost
jobmanager.rpc.port: 6123
jobmanager.bind-host: localhost
jobmanager.memory.process.size: 1600m
jobmanager.execution.failover-strategy: region

taskmanager.bind-host: localhost
taskmanager.host: localhost
taskmanager.memory.process.size: 1728m
taskmanager.numberOfTaskSlots: 8

parallelism.default: 1

fs.allowed-fallback-filesystems: s3
state.backend: rocksdb
execution.checkpointing.interval: 1min
state.checkpoints.dir: <s3p://somebucket/dir/flink-checkpoints>
state.savepoints.dir: <s3p://somebucket/dir/flink-savepoints>
state.backend.incremental: true


rest.address: localhost
rest.bind-address: localhost

clen.moras

03/08/2023, 5:52 PM

have you tried simply with s3 instead of s3p

Open in Slack

Previous Next