Hi everyone, I’m looking for some help. I’m runnin...
# troubleshooting
n
Hi everyone, I’m looking for some help. I’m running
Flink 1.17
with
flink-kubernetes-operator
on EKS and trying to submit a SessionJob with the
jobUri
pointing to an
S3
location. However, I’m encountering an error related to AWS credentials. I believe the IAM roles and S3 bucket permissions are correctly configured, and the service accounts have been properly annotated. and flinkconfig like below
Copy code
kind: FlinkDeployment
spec:
  flinkConfiguration:
    fs.s3.impl: org.apache.hadoop.fs.s3a.S3AFileSystem
    fs.s3a.aws.credentials.provider: com.amazonaws.auth.WebIdentityTokenCredentialsProvider
Error:
Copy code
Caused by: org.apache.hadoop.fs.s3a.auth.NoAuthWithAWSException: No AWS Credentials provided by DynamicTemporaryAWSCredentialsProvider TemporaryAWSCredentialsProvider SimpleAWSCredentialsProvider Environ │
│     at org.apache.hadoop.fs.s3a.AWSCredentialProviderList.getCredentials(AWSCredentialProviderList.java:216)                                                                                                │
│     at com.amazonaws.http.AmazonHttpClient$RequestExecutor.getCredentialsFromContext(AmazonHttpClient.java:1269)                                                                                            │
│     at com.amazonaws.http.AmazonHttpClient$RequestExecutor.runBeforeRequestHandlers(AmazonHttpClient.java:845)                                                                                              │
│     at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:794)                                                                                                             │
│     at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:781)                                                                                                      │
│     at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:755)                                                                                                               │
│     at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:715)                                                                                                            │
│     at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:697)                                                                                                   │
│     at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:561)                                                                                                                               │
│     at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:541)                                                                                                                               │
│     at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5456)                                                                                                                            │
│     at com.amazonaws.services.s3.AmazonS3Client.getBucketRegionViaHeadRequest(AmazonS3Client.java:6432)                                                                                                     │
│     at com.amazonaws.services.s3.AmazonS3Client.fetchRegionFromCache(AmazonS3Client.java:6404)                                                                                                              │
│     at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5441)                                                                                                                            │
│     at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5403)                                                                                                                            │
│     at com.amazonaws.services.s3.AmazonS3Client.getObjectMetadata(AmazonS3Client.java:1372)                                                                                                                 │
│     at org.apache.hadoop.fs.s3a.S3AFileSystem.lambda$getObjectMetadata$10(S3AFileSystem.java:2545)                                                                                                          │
│     at org.apache.hadoop.fs.s3a.Invoker.retryUntranslated(Invoker.java:414)                                                                                                                                 │
│     at org.apache.hadoop.fs.s3a.Invoker.retryUntranslated(Invoker.java:377)                                                                                                                                 │
│     at org.apache.hadoop.fs.s3a.S3AFileSystem.getObjectMetadata(S3AFileSystem.java:2533)                                                                                                                    │
│     at org.apache.hadoop.fs.s3a.S3AFileSystem.getObjectMetadata(S3AFileSystem.java:2513)                                                                                                                    │
│     at org.apache.hadoop.fs.s3a.S3AFileSystem.s3GetFileStatus(S3AFileSystem.java:3776)                                                                                                                      │
│     ... 29 more                                                                                                                                                                                             │
│ Caused by: com.amazonaws.SdkClientException: Unable to load AWS credentials from environment variables (AWS_ACCESS_KEY_ID (or AWS_ACCESS_KEY) and AWS_SECRET_KEY (or AWS_SECRET_ACCESS_KEY))                │
│     at com.amazonaws.auth.EnvironmentVariableCredentialsProvider.getCredentials(EnvironmentVariableCredentialsProvider.java:49)                                                                             │
│     at org.apache.hadoop.fs.s3a.AWSCredentialProviderList.getCredentials(AWSCredentialProviderList.java:177)
Does anyone have suggestions on how to identify and resolve this issue ?
l
Hello, I'm using IRSA. The job jar is download from s3, using an InitContainer. Here is my serviceAccount + FlinkDeployment CR:
Copy code
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: {{ .Values.FlinkDeployment_name }} 
  namespace: {{ .Values.FlinkDeployment_name }}
  annotations:
    eks.amazonaws.com/role-arn:
Copy code
---
apiVersion: flink.apache.org/v1beta1
kind: FlinkDeployment
metadata:
  name: {{ .Values.FlinkDeployment_name }}
spec:
  image: {{ .Values.flink.image }}:{{ .Values.flink.tag }}
  flinkVersion: {{ .Values.flink.flinkVersion }}
  flinkConfiguration:
    execution.checkpointing.savepoint-dir: s3://{{ .Values.FlinkDeployment_name }}/savepoints
    execution.checkpointing.dir: s3://{{ .Values.FlinkDeployment_name }}/checkpoints
    jobmanager.archive.fs.dir: s3://{{ .Values.FlinkDeployment_name }}/archive
    high-availability.storageDir: s3://{{ .Values.FlinkDeployment_name }}/ha
    high-availability.type: kubernetes
  serviceAccount: {{ .Values.FlinkDeployment_name }}
  podTemplate:
    apiVersion: v1
    kind: Pod
    metadata:
      namespace: {{ .Values.FlinkDeployment_name }}
    spec:
      containers:
        - name: flink-main-container
          volumeMounts:
          - mountPath: /flink-data
            name: flink-volume
          - mountPath: /opt/flink/{{ .Values.FlinkDeployment_name }}/artifacts
            name: artifacts
          - mountPath: /.aws
            name: aws
      volumes:
      - name: flink-volume
        emptyDir: {}
      - name: artifacts
        emptyDir: {}
      - name: aws
        emptyDir: {}
  ingress:
    template: "{{ .Values.FlinkDeployment_name }}.{{ .Values.global.labels.region }}.{{ .Values.global.labels.stack }}.talkdesk{{ .Values.ingress.environment }}.{{ .Values.ingress.domain }}"
    className: "{{ .Values.ingressClassName }}"
    tls:
      - hosts:
        - "{{ .Values.FlinkDeployment_name }}.{{ .Values.global.labels.region }}.{{ .Values.global.labels.stack }}.talkdesk{{ .Values.ingress.environment }}.{{ .Values.ingress.domain }}"
        secretName: "{{ .Values.ingress.tls }}"
  jobManager:
    replicas: {{ .Values.flink.jobManager.replicas }}
    podTemplate:
      apiVersion: v1
      kind: Pod
      metadata:
        name: flink-jobmanager
      spec:
        serviceAccount: {{ .Values.FlinkDeployment_name }}
        initContainers:
          - name: awscli
            image: "{{ .Values.awscli.image }}:{{ .Values.awscli.tag }}"
            resources:
              limits:
                cpu: {{ .Values.awscli.limits.cpu }}
                memory: {{ .Values.awscli.limits.memory }}
              requests:
                cpu: {{ .Values.awscli.requests.cpu }}
                memory: {{ .Values.awscli.requests.memory }}
            volumeMounts:
              - mountPath: /opt/flink/{{ .Values.FlinkDeployment_name }}/artifacts
                name: artifacts
              - mountPath: /.aws
                name: aws
            {{- with .Values.securityContext }}
            securityContext:
              runAsUser: {{ .runAsUser }}
              runAsGroup: {{ .runAsGroup }}
            {{- end }}
            command:
            - /bin/sh
            - -c
            args: [
            "sleep 20; \
            aws s3 cp --region {{ .Values.global.labels.region }} s3://td-meza-{{ .Values.global.labels.environment }}-{{ .Values.global.labels.region }}-s3-{{ .Values.flink.bucket }}/{{ .Values.FlinkDeployment_name }}/artifacts/{{ .Values.FlinkDeployment_name }}-{{ .Values.flink.artifacts_version }}.jar /opt/flink/{{ .Values.FlinkDeployment_name }}/artifacts/{{ .Values.FlinkDeployment_name }}-{{ .Values.flink.artifacts_version }}.jar"
            ]
  taskManager:
    replicas: {{ .Values.flink.taskManager.replicas }}
    podTemplate:
      apiVersion: v1
      kind: Pod
      metadata:
        name: flink-taskmanager
      spec:
        serviceAccount: {{ .Values.FlinkDeployment_name }}
        initContainers:
          - name: awscli
            image: "{{ .Values.awscli.image }}:{{ .Values.awscli.tag }}"
            resources:
              limits:
                cpu: {{ .Values.awscli.limits.cpu }}
                memory: {{ .Values.awscli.limits.memory }}
              requests:
                cpu: {{ .Values.awscli.requests.cpu }}
                memory: {{ .Values.awscli.requests.memory }}
            volumeMounts:
              - mountPath: /opt/flink/{{ .Values.FlinkDeployment_name }}/artifacts
                name: artifacts
              - mountPath: /.aws
                name: aws
            {{- with .Values.securityContext }}
            securityContext:
              runAsUser: {{ .runAsUser }}
              runAsGroup: {{ .runAsGroup }}
            {{- end }}
  job:
    jarURI: local:///opt/flink/{{ .Values.FlinkDeployment_name }}/artifacts/{{ .Values.FlinkDeployment_name }}-{{ .Values.flink.artifacts_version }}.jar