Hi All, I'm trying to setup authentication for Had...
# troubleshooting
m
Hi All, I'm trying to setup authentication for Hadoop’s ABFS Azure Filesystem (https://nightlies.apache.org/flink/flink-docs-release-1.20/docs/deployment/filesystems/azure/#abfs) Unfortunately, I don't find an example to store secrets in a safe way, as I don't want to put properties like
fs.azure.account.oauth2.client.secret
directly under
flinkConfiguration
, which generates the ConfigMap related to
flink-conf.yaml
. I thought to do this in the java code, but it seems the configuration is not passed to the task manager, but maybe I am doing something wrong ?
Copy code
Configuration configuration = new Configuration();
configuration.setString("fs.azure.account.auth.type", "OAuth");
configuration.setString("fs.azure.account.oauth.provider.type","org.apache.hadoop.fs.azurebfs.oauth2.ClientCredsTokenProvider");
configuration.setString("fs.azure.account.oauth2.client.id", ...);
configuration.setString("fs.azure.account.oauth2.client.secret", ...);
configuration.setString("fs.azure.account.oauth2.client.endpoint", ...);
FileSystem.initialize(configuration, PluginUtils.createPluginManagerFromRootFolder(configuration));
final StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment(configuration);
env.configure(configuration);
Thanks for your help !