Hi I could not access table on Hive where the file is on s3 Apache Flink #troubleshooting

Hi. I could not access table on Hive where the fil...

Jirawech Siwawut

02/13/2023, 2:46 AM

Hi. I could not access table on Hive where the file is on s3 bucket. Does anyone know where to configure this? I tried this, but it does not work

Copy code

StreamExecutionEnvironment env;
        Properties props = new Properties();
        props.put("s3.access-key", "myaccess");
        props.put("s3.secret-key", "mysecret");
        props.put("s3.endpoint", "myendpoint");
        Configuration conf = ConfigurationUtils.createConfiguration(props);
        env = StreamExecutionEnvironment.createLocalEnvironmentWithWebUI(conf);

Here is the error

Copy code

Caused by: com.amazonaws.AmazonClientException: No AWS Credentials provided by BasicAWSCredentialsProvider EnvironmentVariableCredentialsProvider SharedInstanceProfileCredentialsProvider : com.amazonaws.AmazonClientException: Unable to load credentials from Amazon EC2 metadata service

Gaurav Miglani

02/13/2023, 7:56 AM

try this

Gaurav Miglani

02/13/2023, 7:57 AM

Copy code

System.setProperty("aws.accessKeyId", awsAccessKeyId)
System.setProperty("aws.secretAccessKey", awsSecretAccessKey)

Jirawech Siwawut

02/13/2023, 11:43 AM

It does not work.

Jirawech Siwawut

02/13/2023, 11:44 AM

I need to add hadoop-conf dir and set environment point to hadoop dir. Not sure why s3 config doesnot work with Flink

Reme Ajayi

02/13/2023, 1:56 PM

I'm not using Hive, but things I tried to get s3 to work.

Reme Ajayi

02/13/2023, 1:57 PM

1. putting access keys in

core-site.xml

in the hadoop directory. 2. Setting the

fs.hdfs.hadoopconf

of the my

flink-conf.yaml

to that directory path 3. You can also set your s3 access key and secret in

flink-conf.yaml

but this is not recommended for production

Jirawech Siwawut

02/13/2023, 2:02 PM

Copy code

Properties props = new Properties();
        props.put("fs.hdfs.hadoopconf", "/dev/hadoop-conf");
        Configuration conf = ConfigurationUtils.createConfiguration(props);
        env =

This does not work for me. Setting HADOOP_CONF_DIR does work

Copy code

export HADOOP_CONF_DIR=/dev/hadoop-conf

Reme Ajayi

02/13/2023, 2:16 PM

Can you try setting it in the

flink-conf.yaml

file directly?

Reme Ajayi

02/13/2023, 2:17 PM

Have you put access keys in

core-site.xml

Jirawech Siwawut

02/13/2023, 2:20 PM

Yes. I put access key in

core-site.xml

it works already with this method

Copy code

export HADOOP_CONF_DIR=/dev/hadoop-conf

I will try setting on flink-conf.yaml. I develop in IDE, so i do not know how to do it

2 Views

Open in Slack

Previous Next