hi!
does somebody know how to configure checkpoin...
# troubleshootingd
Dima Sheludko
03/07/2023, 3:47 PMhi!
does somebody know how to configure checkpointing to s3 using presto and aws iam without passing
Copy code
s3.access-key: your-access-key
s3.secret-key: your-secret-keyhive.s3.*s
sap1ens
03/08/2023, 3:27 AMHow is it deployed? I have a similar setup with EKS, one of the node groups has permissions to write to S3 and it’s enough, no need to configure anything else.
c
clen.moras
03/08/2023, 6:02 PMcreate your s3
iam role to connect to said s3 bucket
policy / permissions
oidc auth to essentially limit access , please read up on how to limit access on basis of namespace + service account
flink operator should create a service account by the name "flink" (*default name), in the ns that you are deploying flinkdeployment. Annotate that SA with you iam role.
Update you checkpoint/savepoint/ha related path with appropriate s3 based path. Wait for you reconciler (in case you are using if not simply reapply flinkdeployment manifest).
it should be working.
Since you mentioned s3, i will only speak of aws.
d
Dima Sheludko
03/09/2023, 10:05 AM@sap1ens
thanks for your response!
eks with jobmanager and taskamanger deployments (i am using standalone k8s approach)
service account to access s3
it creates env vars inside pod, but i asked aws support for help and it looks that presto ignore this serviceAcc role and use another one IAM role for our ec2 instances (but if check inside the pod, all AWS env variables are set to the correct role)
i suppose it is possible due to
presto.s3.use-instance-credentialsDima Sheludko
03/09/2023, 11:22 AM@clen.moras thanks for you response
i have single serviceAccount with crud permissions for configmap and it is also annotated with my iam role, the role have all required permissions for s3 bucket, moreover i have checked inside the pod that all requied AWS_ env initialized correctly
but still jobmanager don`t use this iam to communicate with s3 it use our ec2 aim
c
clen.moras
03/09/2023, 11:51 AMin that case it should give you an error when it tried to checkpoint or start with ha options sayin access restriced. what is that error status code?
d
Dima Sheludko
03/09/2023, 11:52 AMexactly, error while triggering checkpoint - 403 forbidden
Dima Sheludko
03/09/2023, 4:53 PMhuh)
fixed, outplayed and destroyed myself with options
maybe it will be useful for somebody
here is code snippet of resolving AwsCredentialsProvider
Copy code
if (credentials.isPresent()) {
return new AWSStaticCredentialsProvider((AWSCredentials)credentials.get());
} else if (this.useInstanceCredentials) {
return InstanceProfileCredentialsProvider.getInstance();
} else if (this.s3IamRole != null) {
return (new STSAssumeRoleSessionCredentialsProvider.Builder(this.s3IamRole, this.s3IamRoleSessionName)).build();
} else {
String providerClass = conf.get("presto.s3.credentials-provider");
return (AWSCredentialsProvider)(!Strings.isNullOrEmpty(providerClass) ? getCustomAWSCredentialsProvider(uri, conf, providerClass) : DefaultAWSCredentialsProviderChain.getInstance());
}3 Views