Hi there. I am using the Flink Kubernetes Operator and used FlinkDeployment to configured checkpoints/savepoints and high availability using S3-compatible storage, Minio. This is working fine. However, I am exploring alternatives to Minio due to its license. I found in the documentation about HDFS support for this. However, I could not find any concrete examples. I have tried using

state.checkpoints.dir: <hdfs://namenode>:port/checkpoints

and I get the following error:

Caused by: org.apache.flink.core.fs.UnsupportedFileSystemSchemeException: Could not find a file system implementation for scheme 'hdfs'. The scheme is not directly supported by Flink and no Hadoop file system to support this scheme could be loaded.

Any ideas? Thanks in advance.

It’s kind of in the error: there is no file system implementation for hdfs in your config. See https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/filesystems/overview/#hadoop-file-system-hdfs-and-its-other-implementations for what’s needed if you want to use hdfs

@Martijn Visser Thanks! I'll check out SeaweedFS.