Hi I was trying to use aws IAM to access s3 but couldn t get Apache Flink #random

Hi, I was trying to use aws IAM to access s3, but...

Zhong Chen

05/08/2023, 4:07 PM

Hi, I was trying to use aws IAM to access s3, but couldn’t get it working. I used the oidc assume role for the service account and passed the iam role as below, but the job was failing to start because it failed to locate the iam role info. What am I missing here?

Copy code

flinkConfiguration:
    taskmanager.numberOfTaskSlots: "2"
    state.checkpoints.dir: <s3://bucket>
    state.savepoints.dir: <s3://bucekt>
    fs.s3a.aws.credentials.provider: com.amazonaws.auth.WebIdentityTokenCredentialsProvider
    fs.s3a.aws.credentials.provider.role.arn: iam arn
  serviceAccount: flink

Zhong Chen

05/08/2023, 4:07 PM

Copy code

Caused by: java.lang.NullPointerException: You must specify a value for roleArn and roleSessionName
	at com.amazonaws.auth.STSAssumeRoleSessionCredentialsProvider$Builder.<init>(STSAssumeRoleSessionCredentialsProvider.java:383) ~[?:?]

Alex T

05/08/2023, 4:13 PM

Is this in EKS? I added the following annotation to the

flink

service account with the IAM role ARN as the value.

<http://eks.amazonaws.com/role-arn|eks.amazonaws.com/role-arn>

Zhong Chen

05/08/2023, 4:51 PM

Is this in EKS?

Yes.

Zhong Chen

05/08/2023, 4:53 PM

How do you do that? The

flink

service account is created by the

k8s flink operator

Zhong Chen

05/08/2023, 4:54 PM

How do you add additional annotation?

Alex T

05/08/2023, 4:56 PM

If you use helm to deploy the operator, you can set the following in the values file

Copy code

jobServiceAccount:
  annotations:
    <http://eks.amazonaws.com/role-arn|eks.amazonaws.com/role-arn>: arn:aws:iam::...

Zhong Chen

05/08/2023, 5:01 PM

cool. thanks!

👍 1

Zhong Chen

05/08/2023, 5:13 PM

we actually use argocd for deploying the helm chart.

Copy code

spec:
      destination:
        name: ''
        namespace: data
      project: k8s-flink-operator
      source:
        repoURL: <https://github.com/apache/flink-kubernetes-operator>
        targetRevision: release-1.4
        path: helm/flink-kubernetes-operator
        helm:
          skipCrds: true
          parameters:
          - name: "webhook.create"
            value: "false"
          - name: "image.repository"
            value: "apache/flink-kubernetes-operator"

Alex T

05/08/2023, 5:46 PM

I deploy from terraform with the following. You may need to escape the

characters in the annotation name in a similar manor.

Copy code

set {
    name  = "jobServiceAccount.annotations.eks\\.amazonaws\\.com/role-arn"
    value = "<role_arn_here>"
  }

👍 1

13 Views

Open in Slack

Previous Next