Hey folks Is Flink 1 16 supporting s3 hadoop irsa on AWS EKS Apache Flink #troubleshooting

Hey folks! Is Flink 1.16 supporting s3 hadoop irsa...

Sylvia Lin

02/16/2023, 1:07 AM

Hey folks! Is Flink 1.16 supporting s3 hadoop irsa on AWS EKS now? Seeing this ticket, but FileSink on s3a still doesn't work with AWS EKS irsa. Any workaround to sink files with s3p? https://issues.apache.org/jira/browse/FLINK-23487?page=com.atlassian.jira.plugin.system.issuetabpanels%3Aall-tabpanel

Sylvia Lin

02/16/2023, 2:09 AM

created a ticket here: https://issues.apache.org/jira/browse/FLINK-31095

sap1ens

02/16/2023, 4:22 AM

Replied, I think the exception is unrelated to permissions.

Sylvia Lin

02/16/2023, 4:25 AM

I think I enabled the plugin, as I logged into the pod and I can see:

Copy code

root@events-translate-649d898f9d-9tzcm:/opt/flink/plugins/s3-fs-hadoop# ls
flink-s3-fs-hadoop-1.16.1.jar

Sylvia Lin

02/16/2023, 4:26 AM

Anything else I need to configure? I followed this doc: https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/filesystems/plugins/

sap1ens

02/16/2023, 4:42 AM

That's pretty much it. Can you try explicitly specifying

s3a

instead of

s3

in the file path?

Sylvia Lin

02/16/2023, 4:43 AM

yes I did add

s3a

to the path too

Sylvia Lin

02/16/2023, 4:45 AM

but few things: 1. the job is not running with default service account, I create a new service account for it 2. the job checkpoint/savepoint/HA data are using

s3p

, which works fine

Sylvia Lin

02/16/2023, 4:47 AM

And from the exception log, seems the caller doesn't pass

s3AccessHelper

down: https://github.com/apache/flink/blob/278dc7b793303d228f7816585054629708983af6/flin[…]/main/java/org/apache/flink/fs/s3/common/FlinkS3FileSystem.java

Sylvia Lin

02/16/2023, 5:44 AM

Ohhh I missed one s3 path which should be

s3a

, it should be the problem. I'm so sorry! Really appreciate your time and help!

🙌 1

Sylvia Lin

02/16/2023, 6:35 AM

Huhhh, new issue comes, now it's a permission issue. But let me verify with AWS first

Sylvia Lin

02/16/2023, 4:54 PM

Hey Yaroslav, we verified the permission issue is caused by FileSink made the request based on node group iam role, but not the iam role we binding to the service account. For EKS, any special hadoop config required to build the job image?

Copy code

User Agent: Hadoop 3.3.2, aws-sdk-java/1.11.951 Linux/5.4.209-116.367.amzn2.x86_64 OpenJDK_64-Bit_Server_VM/25.362-b09 java/1.8.0_362 vendor/Temurin

Requester: arn:aws:iam::***:role/second_group-eks-node-group-***

Response Code: 403

Error Code: AccessDenied

sap1ens

02/16/2023, 4:57 PM

Sorry, not sure, I’m just relying on node group roles.

Sylvia Lin

02/16/2023, 4:58 PM

Oh OK, thank you! We suppose this should use the iam role we binding to the service account.

Hygor Knust

04/19/2023, 5:07 PM

Hi Sylvia, are you using the

flink-kubernetes-operator

with the

FlinkSessionJob

CRD?

Sylvia Lin

04/19/2023, 6:14 PM

no, all application deployment

Hygor Knust

04/19/2023, 6:17 PM

I see, thank you

Sylvia Lin

04/19/2023, 6:18 PM

btw Martijn suggested to add

fs.s3a.aws.credentials.provider: com.amazonaws.auth.WebIdentityTokenCredentialsProvider

, which should work.

Hygor Knust

04/19/2023, 6:42 PM

Yes, this configurations works on an Application Deployment. But when using a SessionJob the

flink-kuberenetes-operator

Pod itself is responsible for fetching the JAR file. In my case this JAR file is stored in S3, hence the need for the operator pod to access S3. In the Helm values for the operator there is no way to configure that. https://nightlies.apache.org/flink/flink-kubernetes-operator-docs-main/docs/operations/helm/ I tried this but had no luck:

Copy code

jvmArgs:
  operator: "-Daws.credentials.provider=com.amazonaws.auth.WebIdentityTokenCredentialsProvider -Daws.webIdentityTokenFile=/var/run/secrets/eks.amazonaws.com/serviceaccount/token"

Open in Slack

Previous Next