Hello folks ! I'm trying to have flink manage che...
# troubleshooting
j
Hello folks ! I'm trying to have flink manage checkpoints/savepoints on S3, using my current pod AWS IAM role. Here is the config i'm using :
Copy code
state.checkpoints.dir=s3://my/path
 state.savepoints.dir=s3://my/path
 fs.allowed-fallback-filesystems=s3
 presto.s3.use-instance-credentials=false
I am running flink 1.19 using the docker image. I keep getting the following error (the trace is longer, but I think this is the relevant part) :
Copy code
Caused by: java.lang.RuntimeException: S3 credentials not configured
	at org.apache.flink.fs.s3presto.shaded.com.facebook.presto.hive.s3.PrestoS3FileSystem.getAwsCredentialsProvider(PrestoS3FileSystem.java:711)
	at org.apache.flink.fs.s3presto.shaded.com.facebook.presto.hive.s3.PrestoS3FileSystem.createAmazonS3Client(PrestoS3FileSystem.java:632)
	at org.apache.flink.fs.s3presto.shaded.com.facebook.presto.hive.s3.PrestoS3FileSystem.initialize(PrestoS3FileSystem.java:216)
	at org.apache.flink.fs.s3.common.AbstractS3FileSystemFactory.create(AbstractS3FileSystemFactory.java:126)
As far as I can tell from this trace, and my config, I should end up here and the
DefaultAWSCredentialsProviderChain
should do the job for me (i've tested just that func call and this works) : I have not iam role presto config key set, nor any custom provider class. Does the error I get mean anything to someone ? Am I missing more config entries to get it right (some presto/hive entries maybe ? Thanks a lot for your help !
βœ… 1
s
If you are using service account OIDC approach for IAM access. Note that I am not using hive/presto so it could be different case for you.
You can try adding below in flinkConfiguration
fs.s3a.aws.credentials.provider: com.amazonaws.auth.WebIdentityTokenCredentialsProvider
βž• 1
j
hello @Sushant, thank you for your input. I'm now trying to use hadoop, so I added the
flink-s3-fs-hadoop-1.9.3.jar
to
plugins/s3/
in my Flink install. I also added
aws-java-sdk-core-1.9.9.jar
in
plugins/s3/
Then, adding
fs.s3a.aws.credentials.provider: com.amazonaws.auth.WebIdentityTokenCredentialsProvider
I get the following error :
Copy code
java.io.IOException: From option fs.s3a.aws.credentials.provider java.lang.ClassNotFoundException: Class org.apache.flink.fs.s3base.shaded.com.amazonaws.auth.WebIdentityTokenCredentialsProvider not found
I tried to follow the latest doc. Thx again for your help !
I think flink is not loading Hadoop, I see this trace, when the jobmanager starts :
Copy code
Install default filesystem.
2024-06-17 14:19:47,095 INFO  org.apache.flink.core.fs.FileSystem                          [] - Hadoop is not in the classpath/dependencies. The extended set of supported File Systems via Hadoop is not available.
hey ! I've tried moving both aws-skd-core and flink-hadoop jars to
plugins/s3
,
plugins/s3-fs-hadoop
and even
flink/lib
, but this does not seem to take effect : it still canot find the aws provider class. Could you detail how you set up hadoop on your flink install please,, I must be close ! πŸ™‚ Thx πŸ™
s
I have flink as docker image , below are three essential commands for flink 1.18.1:
RUN mkdir -p $FLINK_HOME/usrlib
RUN mkdir -p $FLINK_HOME/plugins/s3-fs-hadoop/
RUN wget https://repo1.maven.org/maven2/org/apache/flink/flink-s3-fs-hadoop/1.18.1/flink-s3-fs-hadoop-1.18.1.jar -P $FLINK_HOME/plugins/s3-fs-hadoop/
j
thx , I have the same πŸ‘ Do you also use a specific AWSProvider class ? If yes, where did you add the jar ?
typical silly mistake, I was not matching the hadoop flink jar to my Flink version πŸ€¦β€β™‚οΈ The path you posted made me realize it, it is now ok, thank you for your help πŸ™
s
Glad it worked for you