https://flink.apache.org/ logo
Join Slack
Powered by
:wave: I'm trying to make jobs in my FlinkDeployme...
# troubleshooting
j
πŸ‘‹ I'm trying to make jobs in my FlinkDeployment stateful using the information here in the documentation. I'm running Flink using the operator in Kubernetes (EKS) and my first idea was to use an EBS-backed PVC. It seems that because the jobManager and taskManager run on different pods I would have to configure the PVC as 
ReadWriteMany
, which is not supported for EBS volumes. Can anyone point me in the right direction? I see that EFS might be an option, but I'm curious what the recommended options are for configuring savepoint/checkpoint and ha volumes for Flink jobManagers in EKS. I'll put my current configuration in a thread
Copy code
high-availability: org.apache.flink.kubernetes.highavailability.KubernetesHaServicesFactory
    high-availability.storageDir: file:///flink-data/ha
    state.savepoints.dir: file:///flink-data/savepoints
    state.checkpoints.dir: <file://flink-data/checkpoints>
I see there might be a way to use S3, but I would rather use a PVC that's mounted locally if I can
After some more reading I'm starting to lean toward using S3 for storage. I'll update this with any discoveries as I go along πŸ€
I'm using the 
flink-s3-fs-hadoop
plugin and my configuration looks like this:
Copy code
flinkConfiguration:
    high-availability: org.apache.flink.kubernetes.highavailability.KubernetesHaServicesFactory
    high-availability.storageDir: <s3://staging-flink-poc/flink-data/ha>
    state.savepoints.dir: <s3://staging-flink-poc/flink-data/savepoints>
    state.checkpoints.dir: <s3://staging-flink-poc/flink-data/checkpoints>
I created an S3 bucket and set it up for IRSA. The service account it should have permissions for is named 
flink
. I've set up IRSA many times in my org so I know that side of things is properly configured. I have this set in my `FlinkDeployment`:
Copy code
spec:
...
  serviceAccount: flink
And I see that the sa exists in the namespace and that it has the appropriate role associated:
Copy code
k get sa flink -o yaml
apiVersion: v1
...
kind: ServiceAccount
metadata:
  annotations:
    <http://eks.amazonaws.com/role-arn|eks.amazonaws.com/role-arn>: arn:aws:iam::xxxxxxxxxxx:role/staging-flink-poc
...
When I try this configuration, I see that the jobmanager is trying to use a different role, related to the node it's on:
Copy code
Caused by: com.amazonaws.services.s3.model.AmazonS3Exception: User: arn:aws:sts::xxxxxxxxxxx:assumed-role/karpenter-node/i-xxxxxxxxxxx is not authorized to perform: s3:ListBucket on resource: "arn:aws:s3:::staging-flink-poc" because no identity-based policy allows the s3:ListBucket action (Service: Amazon S3; Status Code: 403; Error Code: AccessDenied; ...
I see in the Pod itself is using the service account I specified and it even has the IAM role reference it should have:
Copy code
apiVersion: v1
kind: Pod
...
spec:
  containers:
...
    env:
...
    - name: AWS_ROLE_ARN
      value: arn:aws:iam::xxxxxxxxxxx:role/staging-flink-poc
    - name: AWS_WEB_IDENTITY_TOKEN_FILE
      value: /var/run/secrets/eks.amazonaws.com/serviceaccount/token
...
    name: flink-main-container
...
  serviceAccount: flink
  serviceAccountName: flink
I'm using the 
flink-s3-fs-hadoop
plugin and my configuration looks like this:
Copy code
flinkConfiguration:
    high-availability: org.apache.flink.kubernetes.highavailability.KubernetesHaServicesFactory
    high-availability.storageDir: <s3://staging-flink-poc/flink-data/ha>
    state.savepoints.dir: <s3://staging-flink-poc/flink-data/savepoints>
    state.checkpoints.dir: <s3://staging-flink-poc/flink-data/checkpoints>
I fixed this by setting:
Copy code
fs.s3a.aws.credentials.provider: com.amazonaws.auth.WebIdentityTokenCredentialsProvider
in my configuration. I'm getting a different error now, but that's progress!
Open in Slack
PreviousNext
Powered by