Jean-Benoit Hamard
06/16/2024, 6:44 PMstate.checkpoints.dir=s3://my/path
state.savepoints.dir=s3://my/path
fs.allowed-fallback-filesystems=s3
presto.s3.use-instance-credentials=false
I am running flink 1.19 using the docker image.
I keep getting the following error (the trace is longer, but I think this is the relevant part) :
Caused by: java.lang.RuntimeException: S3 credentials not configured
at org.apache.flink.fs.s3presto.shaded.com.facebook.presto.hive.s3.PrestoS3FileSystem.getAwsCredentialsProvider(PrestoS3FileSystem.java:711)
at org.apache.flink.fs.s3presto.shaded.com.facebook.presto.hive.s3.PrestoS3FileSystem.createAmazonS3Client(PrestoS3FileSystem.java:632)
at org.apache.flink.fs.s3presto.shaded.com.facebook.presto.hive.s3.PrestoS3FileSystem.initialize(PrestoS3FileSystem.java:216)
at org.apache.flink.fs.s3.common.AbstractS3FileSystemFactory.create(AbstractS3FileSystemFactory.java:126)
As far as I can tell from this trace, and my config, I should end up here and the DefaultAWSCredentialsProviderChain should do the job for me (i've tested just that func call and this works) : I have not iam role presto config key set, nor any custom provider class.
Does the error I get mean anything to someone ?
Am I missing more config entries to get it right (some presto/hive entries maybe ?
Thanks a lot for your help !Sushant
06/17/2024, 11:07 AMSushant
06/17/2024, 11:07 AMfs.s3a.aws.credentials.provider: com.amazonaws.auth.WebIdentityTokenCredentialsProviderJean-Benoit Hamard
06/17/2024, 2:08 PMflink-s3-fs-hadoop-1.9.3.jar to plugins/s3/ in my Flink install.
I also added aws-java-sdk-core-1.9.9.jar in plugins/s3/
Then, adding fs.s3a.aws.credentials.provider: com.amazonaws.auth.WebIdentityTokenCredentialsProvider
I get the following error :
java.io.IOException: From option fs.s3a.aws.credentials.provider java.lang.ClassNotFoundException: Class org.apache.flink.fs.s3base.shaded.com.amazonaws.auth.WebIdentityTokenCredentialsProvider not found
I tried to follow the latest doc. Thx again for your help !Jean-Benoit Hamard
06/17/2024, 2:30 PMInstall default filesystem.
2024-06-17 14:19:47,095 INFO org.apache.flink.core.fs.FileSystem [] - Hadoop is not in the classpath/dependencies. The extended set of supported File Systems via Hadoop is not available.Jean-Benoit Hamard
06/18/2024, 9:59 AMplugins/s3, plugins/s3-fs-hadoop and even flink/lib, but this does not seem to take effect : it still canot find the aws provider class.
Could you detail how you set up hadoop on your flink install please,, I must be close ! π Thx πSushant
06/18/2024, 10:06 AMRUN mkdir -p $FLINK_HOME/usrlib
RUN mkdir -p $FLINK_HOME/plugins/s3-fs-hadoop/ RUN wget https://repo1.maven.org/maven2/org/apache/flink/flink-s3-fs-hadoop/1.18.1/flink-s3-fs-hadoop-1.18.1.jar -P $FLINK_HOME/plugins/s3-fs-hadoop/Jean-Benoit Hamard
06/18/2024, 11:53 AMJean-Benoit Hamard
06/18/2024, 12:59 PMSushant
06/18/2024, 1:33 PM