Hello, I would like to ask the communities a question about AWS credentials. At my company, we are employing an On-Prem Flink app, which upload data records to AWS Kinesis Stream. The problem is that our app is designed to be long-running, but the security policies of On-Prem environment mandate session credential tokens, which expire at fixed interval (lets say 30 mins). This credential providers are loaded once when Flink build the job graph, and never refresh (according to the shaded implementation in 1.15.2). Hence, every 30 minutes, our Flink app hangs because Kinesis writers keep retrying while receiving Expired Token Exception (founded in the Debug log). This blocks all the record sending to KDS till the Job Manager restarts all due to checkpoint writing exception. Therefore, I would like to ask if we can have another AWS credential setup to handle this short-expiring credentials? Thanks

Hello, we can certainly work something out. For your usecase how are you rotating the creds? Which credential provider are you currently using?

Thanks for your reply, we are using ProfileCredentialsProvider, which loads secret, access key, and session token from .aws/credential file. This file is constantly updated by the AWS SSM agent.

Are you using EFO?

About the ProfileCredentialProvider, we spent time debugging and found that it keeps using the same token content and never reloads. On the other hand, the Kinesis Sink Writer does not consider this as fatal exception. Our app had its availability extremely low due to the timeout (the expiration is 1 hour, but timeout 10 minutes plus the restart time)

Ok got it. It is a shame AWS SDK dropped support for the reload. I will follow up tomorrow as not at laptop atm.

Thanks. We currently have a workaround by customizing the writer to detect the expired token exception and use java reflection to twist the token object.

cc @G

Related https://github.com/aws/aws-sdk-java-v2/issues/1754

Thanks, glad to know that AWS is fixing this problem.

I am not sure if AWS are fixing it, the issue looks stale. The conversation implies the feature was dropped intentionally as it was not supported for other language SDK. We will need to do something, I am looking forward to hearing more about this FLIP from @G 😊

Hi All, thanks for pinging me! Yeah, today is the planned day to open the

Generalized delegation token support

FLIP which allows non-hadoop token provider implementations in the near future 🙂

Thanks for the references @G. @Tri Tam Hoang given you have a workaround I propose we wait for the new feature and build on top of this. However, it would be fairly straightforward to implement a wrapper AwsCredentialProvider that delegates with refresh if we need to

@Danny Cranmer thanks for the heads up. Btw, there is a AWS fix linked to the https://github.com/aws/aws-sdk-java-v2/issues/1754. The development is still active.

Yes another option is that it is fixed upstream. However this PR you mention was opened in 2020, I will try to get an update.