Niro
03/27/2025, 2:16 PMFlink 1.17 with flink-kubernetes-operator on EKS and trying to submit a SessionJob with the jobUri pointing to an S3 location.
However, I’m encountering an error related to AWS credentials. I believe the IAM roles and S3 bucket permissions are correctly configured, and the service accounts have been properly annotated.
and flinkconfig like below
kind: FlinkDeployment
spec:
flinkConfiguration:
fs.s3.impl: org.apache.hadoop.fs.s3a.S3AFileSystem
fs.s3a.aws.credentials.provider: com.amazonaws.auth.WebIdentityTokenCredentialsProvider
Error:
Caused by: org.apache.hadoop.fs.s3a.auth.NoAuthWithAWSException: No AWS Credentials provided by DynamicTemporaryAWSCredentialsProvider TemporaryAWSCredentialsProvider SimpleAWSCredentialsProvider Environ │
│ at org.apache.hadoop.fs.s3a.AWSCredentialProviderList.getCredentials(AWSCredentialProviderList.java:216) │
│ at com.amazonaws.http.AmazonHttpClient$RequestExecutor.getCredentialsFromContext(AmazonHttpClient.java:1269) │
│ at com.amazonaws.http.AmazonHttpClient$RequestExecutor.runBeforeRequestHandlers(AmazonHttpClient.java:845) │
│ at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:794) │
│ at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:781) │
│ at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:755) │
│ at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:715) │
│ at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:697) │
│ at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:561) │
│ at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:541) │
│ at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5456) │
│ at com.amazonaws.services.s3.AmazonS3Client.getBucketRegionViaHeadRequest(AmazonS3Client.java:6432) │
│ at com.amazonaws.services.s3.AmazonS3Client.fetchRegionFromCache(AmazonS3Client.java:6404) │
│ at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5441) │
│ at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5403) │
│ at com.amazonaws.services.s3.AmazonS3Client.getObjectMetadata(AmazonS3Client.java:1372) │
│ at org.apache.hadoop.fs.s3a.S3AFileSystem.lambda$getObjectMetadata$10(S3AFileSystem.java:2545) │
│ at org.apache.hadoop.fs.s3a.Invoker.retryUntranslated(Invoker.java:414) │
│ at org.apache.hadoop.fs.s3a.Invoker.retryUntranslated(Invoker.java:377) │
│ at org.apache.hadoop.fs.s3a.S3AFileSystem.getObjectMetadata(S3AFileSystem.java:2533) │
│ at org.apache.hadoop.fs.s3a.S3AFileSystem.getObjectMetadata(S3AFileSystem.java:2513) │
│ at org.apache.hadoop.fs.s3a.S3AFileSystem.s3GetFileStatus(S3AFileSystem.java:3776) │
│ ... 29 more │
│ Caused by: com.amazonaws.SdkClientException: Unable to load AWS credentials from environment variables (AWS_ACCESS_KEY_ID (or AWS_ACCESS_KEY) and AWS_SECRET_KEY (or AWS_SECRET_ACCESS_KEY)) │
│ at com.amazonaws.auth.EnvironmentVariableCredentialsProvider.getCredentials(EnvironmentVariableCredentialsProvider.java:49) │
│ at org.apache.hadoop.fs.s3a.AWSCredentialProviderList.getCredentials(AWSCredentialProviderList.java:177)
Does anyone have suggestions on how to identify and resolve this issue ?Luís Costa
04/09/2025, 5:33 PM---
apiVersion: v1
kind: ServiceAccount
metadata:
name: {{ .Values.FlinkDeployment_name }}
namespace: {{ .Values.FlinkDeployment_name }}
annotations:
eks.amazonaws.com/role-arn:
---
apiVersion: flink.apache.org/v1beta1
kind: FlinkDeployment
metadata:
name: {{ .Values.FlinkDeployment_name }}
spec:
image: {{ .Values.flink.image }}:{{ .Values.flink.tag }}
flinkVersion: {{ .Values.flink.flinkVersion }}
flinkConfiguration:
execution.checkpointing.savepoint-dir: s3://{{ .Values.FlinkDeployment_name }}/savepoints
execution.checkpointing.dir: s3://{{ .Values.FlinkDeployment_name }}/checkpoints
jobmanager.archive.fs.dir: s3://{{ .Values.FlinkDeployment_name }}/archive
high-availability.storageDir: s3://{{ .Values.FlinkDeployment_name }}/ha
high-availability.type: kubernetes
serviceAccount: {{ .Values.FlinkDeployment_name }}
podTemplate:
apiVersion: v1
kind: Pod
metadata:
namespace: {{ .Values.FlinkDeployment_name }}
spec:
containers:
- name: flink-main-container
volumeMounts:
- mountPath: /flink-data
name: flink-volume
- mountPath: /opt/flink/{{ .Values.FlinkDeployment_name }}/artifacts
name: artifacts
- mountPath: /.aws
name: aws
volumes:
- name: flink-volume
emptyDir: {}
- name: artifacts
emptyDir: {}
- name: aws
emptyDir: {}
ingress:
template: "{{ .Values.FlinkDeployment_name }}.{{ .Values.global.labels.region }}.{{ .Values.global.labels.stack }}.talkdesk{{ .Values.ingress.environment }}.{{ .Values.ingress.domain }}"
className: "{{ .Values.ingressClassName }}"
tls:
- hosts:
- "{{ .Values.FlinkDeployment_name }}.{{ .Values.global.labels.region }}.{{ .Values.global.labels.stack }}.talkdesk{{ .Values.ingress.environment }}.{{ .Values.ingress.domain }}"
secretName: "{{ .Values.ingress.tls }}"
jobManager:
replicas: {{ .Values.flink.jobManager.replicas }}
podTemplate:
apiVersion: v1
kind: Pod
metadata:
name: flink-jobmanager
spec:
serviceAccount: {{ .Values.FlinkDeployment_name }}
initContainers:
- name: awscli
image: "{{ .Values.awscli.image }}:{{ .Values.awscli.tag }}"
resources:
limits:
cpu: {{ .Values.awscli.limits.cpu }}
memory: {{ .Values.awscli.limits.memory }}
requests:
cpu: {{ .Values.awscli.requests.cpu }}
memory: {{ .Values.awscli.requests.memory }}
volumeMounts:
- mountPath: /opt/flink/{{ .Values.FlinkDeployment_name }}/artifacts
name: artifacts
- mountPath: /.aws
name: aws
{{- with .Values.securityContext }}
securityContext:
runAsUser: {{ .runAsUser }}
runAsGroup: {{ .runAsGroup }}
{{- end }}
command:
- /bin/sh
- -c
args: [
"sleep 20; \
aws s3 cp --region {{ .Values.global.labels.region }} s3://td-meza-{{ .Values.global.labels.environment }}-{{ .Values.global.labels.region }}-s3-{{ .Values.flink.bucket }}/{{ .Values.FlinkDeployment_name }}/artifacts/{{ .Values.FlinkDeployment_name }}-{{ .Values.flink.artifacts_version }}.jar /opt/flink/{{ .Values.FlinkDeployment_name }}/artifacts/{{ .Values.FlinkDeployment_name }}-{{ .Values.flink.artifacts_version }}.jar"
]
taskManager:
replicas: {{ .Values.flink.taskManager.replicas }}
podTemplate:
apiVersion: v1
kind: Pod
metadata:
name: flink-taskmanager
spec:
serviceAccount: {{ .Values.FlinkDeployment_name }}
initContainers:
- name: awscli
image: "{{ .Values.awscli.image }}:{{ .Values.awscli.tag }}"
resources:
limits:
cpu: {{ .Values.awscli.limits.cpu }}
memory: {{ .Values.awscli.limits.memory }}
requests:
cpu: {{ .Values.awscli.requests.cpu }}
memory: {{ .Values.awscli.requests.memory }}
volumeMounts:
- mountPath: /opt/flink/{{ .Values.FlinkDeployment_name }}/artifacts
name: artifacts
- mountPath: /.aws
name: aws
{{- with .Values.securityContext }}
securityContext:
runAsUser: {{ .runAsUser }}
runAsGroup: {{ .runAsGroup }}
{{- end }}
job:
jarURI: local:///opt/flink/{{ .Values.FlinkDeployment_name }}/artifacts/{{ .Values.FlinkDeployment_name }}-{{ .Values.flink.artifacts_version }}.jar