Hello, does anyone know how to pass AWS configs to flink's flink-s3-fs-hadoop plugin during local testing? I made this stack overflow post with more information: https://stackoverflow.com/questions/77540049/how-can-i-pass-aws-configs-to-flinks-flink-s3-fs-hadoop-plugin-in-local-tests

Hi, have you tried to specify hadoop s3 configuration? https://hadoop.apache.org/docs/stable/hadoop-aws/tools/hadoop-aws/index.html#General_S3A_Client_configuration

Yup, I've tried to specify that using many different methods; the problem is that the plugin doesn't seem to be receiving those configs Note: full details of what I've tried (incl. the above) are in the stackoverflow post

You could try this: 1- Create your test config file (flink-config.yaml) with

## S3 config
s3.access-key: test
s3.secret-key: test
s3.endpoint: <http://awslocal:4566>
s3.endpoint.region: us-east-1
s3.path.style.access: true

2- Load this configuration with

import org.apache.flink.configuration.Configuration;
import org.apache.flink.configuration.GlobalConfiguration;
import org.apache.flink.core.fs.FileSystem;
import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;

Configuration loadedConfig = GlobalConfiguration.loadConfiguration("directory/where/is/flinkconfig");
FileSystem.initialize(config, null);
StreamExecutionEnvironment environment = StreamExecutionEnvironment.createLocalEnvironmentWithWebUI(loadedConfig);

The important part is

FileSystem.initialize(config, null);

Thank you kindly; it worked!