https://datahubproject.io logo
Join Slack
Powered by
Hello All! We've got most things in our deploymen...
# all-things-deployment
g
Hello All! We've got most things in our deployment sorted out, but stuck on one last bug with fetching AWS credentials. I know a couple other people have deployed using mostly managed AWS services, so wanted to see if anyone else has run into the same issues as us and how you solved it. What we're seeing: Calls to AWS Glue to fetch schemas seems to fail with credentials not found(See log in thread). However, if I ssh into the pod, I am able to run glue fetch schemas, get current role, etc with the CLI. So the role seems valid, and the latest version of AWS cli seems to pick up the role just fine. The Glue and AWS dependency seem pretty up to date. Any idea on what else could be causing this issue? We've seen this type of issue before when the AWS library is out of data, and our workaround was to have a background thread that refreshes credentials to a credential file. So we can use that same solution, but ofc would prefer not to.
Copy code
Caused by: software.amazon.awssdk.core.exception.SdkClientException: Unable to load credentials from any of the providers in the chain AwsCredentialsProviderChain(credentialsProviders=[SystemPropertyCredentialsProvider(), EnvironmentVariableCredentialsProvider(), WebIdentityTokenCredentialsProvider(), ProfileCredentialsProvider(), ContainerCredentialsProvider(), InstanceProfileCredentialsProvider()]) : [SystemPropertyCredentialsProvider(): Unable to load credentials from system settings. Access key must be specified either via environment variable (AWS_ACCESS_KEY_ID) or system property (aws.accessKeyId)., EnvironmentVariableCredentialsProvider(): Unable to load credentials from system settings. Access key must be specified either via environment variable (AWS_ACCESS_KEY_ID) or system property (aws.accessKeyId)., WebIdentityTokenCredentialsProvider(): Multiple HTTP implementations were found on the classpath. To avoid non-deterministic loading implementations, please explicitly provide an HTTP client via the client builders, set the software.amazon.awssdk.http.service.impl system property with the FQCN of the HTTP service to use as the default, or remove all but one HTTP implementation from the classpath, ProfileCredentialsProvider(): Profile file contained no credentials for profile 'default': ProfileFile(profiles=[]), ContainerCredentialsProvider(): Cannot fetch credentials from container - neither AWS_CONTAINER_CREDENTIALS_FULL_URI or AWS_CONTAINER_CREDENTIALS_RELATIVE_URI environment variables are set., InstanceProfileCredentialsProvider(): The requested metadata is not found at <http://169.254.169.254/latest/meta-data/iam/security-credentials/>]
From reading the AWS docs the DefaultCredentialProvider should be able to pickup the IAM role
b
Ah this is instead of providing access key / secret as variables inside of the recipe?
g
I'm unfamiliar with what recipe means, but this occurs when the datahub-gms service tries to ingest anything. But yes, if a key/secret isn't in the env variables for datahub-gms, ingestion fails. But if we set temp creds(secret_key, secret_id, token), ingestion starts working
b
Recipe just meaning the YML configuration for your ingestion!
g
Ah gotcha, thanks for the clarification! So there's no Recipe involved. We are using the Python REST API to emit dataset aspects. The failure is when the gms service is ingesting after receiving a request. The failure looks to be coming from this code: https://github.com/linkedin/datahub/blob/27a5f6dc3b2339e853a0d2dd0956a6a1bb8a10c5/[…]/factory/kafka/schemaregistry/AwsGlueSchemaRegistryFactory.java but nothing in that code is suspicious in any way. Here's the stack trace if it's useful:
Copy code
at software.amazon.awssdk.core.exception.SdkClientException$BuilderImpl.build(SdkClientException.java:98)
	at software.amazon.awssdk.auth.credentials.AwsCredentialsProviderChain.resolveCredentials(AwsCredentialsProviderChain.java:112)
	at software.amazon.awssdk.auth.credentials.internal.LazyAwsCredentialsProvider.resolveCredentials(LazyAwsCredentialsProvider.java:45)
	at software.amazon.awssdk.auth.credentials.DefaultCredentialsProvider.resolveCredentials(DefaultCredentialsProvider.java:104)
	at software.amazon.awssdk.awscore.client.handler.AwsClientHandlerUtils.createExecutionContext(AwsClientHandlerUtils.java:79)
	at software.amazon.awssdk.awscore.client.handler.AwsSyncClientHandler.createExecutionContext(AwsSyncClientHandler.java:68)
	at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.lambda$execute$1(BaseSyncClientHandler.java:99)
	at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.measureApiCallSuccess(BaseSyncClientHandler.java:169)
	at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.execute(BaseSyncClientHandler.java:95)
	at software.amazon.awssdk.core.client.handler.SdkSyncClientHandler.execute(SdkSyncClientHandler.java:45)
	at software.amazon.awssdk.awscore.client.handler.AwsSyncClientHandler.execute(AwsSyncClientHandler.java:55)
	at software.amazon.awssdk.services.glue.DefaultGlueClient.getSchemaByDefinition(DefaultGlueClient.java:6837)
	at com.amazonaws.services.schemaregistry.common.AWSSchemaRegistryClient.getSchemaVersionIdByDefinition(AWSSchemaRegistryClient.java:134)
b
Oh i see - thank you!
If you log into that container, and attempt to echo the 
AWS_ACCESS_KEY_ID
what do you get?
g
AWS_ACCESS_KEY_ID
shouldn't need to be set afaik, we don't use it when calling into any other AWS services(S3, KMS, etc.) And this maybe what's different about the Glue code we use the IAM env variables: https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-role.html
Copy code
bash-5.1$ env | grep AWS
AWS_DEFAULT_REGION=xxxxxx
AWS_REGION=xxxxxxx
AWS_ROLE_ARN=arn:aws:iam::00000000:role/xxxxxx
AWS_WEB_IDENTITY_TOKEN_FILE=/xxxxxxxx
n
hi! did you manage to solve this? I am facing this exact error
Copy code
Caused by: software.amazon.awssdk.core.exception.SdkClientException: Unable to load credentials from any of the providers in the chain AwsCredentialsProviderChain(credentialsProviders=[SystemPropertyCredentialsProvider(), EnvironmentVariableCredentialsProvider(), WebIdentityTokenCredentialsProvider(), ProfileCredentialsProvider(), ContainerCredentialsProvider(), InstanceProfileCredentialsProvider()]) : [SystemPropertyCredentialsProvider(): Unable to load credentials from system settings. Access key must be specified either via environment variable (AWS_ACCESS_KEY_ID) or system property (aws.accessKeyId)., EnvironmentVariableCredentialsProvider(): Unable to load credentials from system settings. Access key must be specified either via environment variable (AWS_ACCESS_KEY_ID) or system property (aws.accessKeyId)., WebIdentityTokenCredentialsProvider(): Either the environment variable AWS_WEB_IDENTITY_TOKEN_FILE or the javaproperty aws.webIdentityTokenFile must be set., ProfileCredentialsProvider(): Profile file contained no credentials for profile 'default': ProfileFile(profiles=[]), ContainerCredentialsProvider(): Cannot fetch credentials from container - neither AWS_CONTAINER_CREDENTIALS_FULL_URI or AWS_CONTAINER_CREDENTIALS_RELATIVE_URI environment variables are set., InstanceProfileCredentialsProvider(): Unable to load credentials from service endpoint.]
	at software.amazon.awssdk.core.exception.SdkClientException$BuilderImpl.build(SdkClientException.java:98)
	at software.amazon.awssdk.auth.credentials.AwsCredentialsProviderChain.resolveCredentials(AwsCredentialsProviderChain.java:112)
	at software.amazon.awssdk.auth.credentials.internal.LazyAwsCredentialsProvider.resolveCredentials(LazyAwsCredentialsProvider.java:45)
	at software.amazon.awssdk.auth.credentials.DefaultCredentialsProvider.resolveCredentials(DefaultCredentialsProvider.java:104)
	at software.amazon.awssdk.awscore.client.handler.AwsClientHandlerUtils.createExecutionContext(AwsClientHandlerUtils.java:79)
	at software.amazon.awssdk.awscore.client.handler.AwsSyncClientHandler.createExecutionContext(AwsSyncClientHandler.java:68)
	at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.lambda$execute$1(BaseSyncClientHandler.java:99)
	at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.measureApiCallSuccess(BaseSyncClientHandler.java:169)
	at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.execute(BaseSyncClientHandler.java:95)
	at software.amazon.awssdk.core.client.handler.SdkSyncClientHandler.execute(SdkSyncClientHandler.java:45)
	at software.amazon.awssdk.awscore.client.handler.AwsSyncClientHandler.execute(AwsSyncClientHandler.java:55)
	at software.amazon.awssdk.services.glue.DefaultGlueClient.getSchemaByDefinition(DefaultGlueClient.java:6837)
	at com.amazonaws.services.schemaregistry.common.AWSSchemaRegistryClient.getSchemaVersionIdByDefinition(AWSSchemaRegistryClient.java:134)
	... 50 common frames omitted
g
I could not figure out what was the issue here. Here were the things i tried to see if an alternative solution worked: 1. Wrote to ~/.aws credentials with temporary credentials. I was hoping to maybe run a background thread to refresh creds at some cadends. That didn't work 2. Bumped up the version of Glue dependency to see if there was an issue with the AWS package being out of date - Didn't work The ROI for this was starting to be small, so I ended up creating a user and adding in the static creds of the user into the environment variables on startup(Awful, I know 😞 )
When I logged into the pod, I was able to call Glue using the AWS cli and inspect the schema, so clearly the role was working for the pod, but for some reason the glue-schema-regisry was not working
and in the stack trace you can clearly see that 
WebIdentityTokenCredentialsProvider
is being attempted, but somehow not finding the pod role
n
thank you @gorgeous-dinner-4055, would be awesome if someone could look into this
@little-megabyte-1074 can you help us please?
w
Would be great if the aws credentials could be read from the kuberentes pod. Would provide an alternative to hard coding the credentials in recipe
n
@gorgeous-dinner-4055 just for you to know
I was able to bypass this by setting explicitly the implementation than I want to use
something like
Copy code
extraEnvs:
    - name: JAVA_OPTS
      value: "-Dsoftware.amazon.awssdk.http.service.impl=software.amazon.awssdk.http.urlconnection.UrlConnectionSdkHttpService"
in the values file of the datahub-gms
did the trick
g
Oh, this is awesome, thanks so much @numerous-camera-74294!!! I'll give that a try!
Just circling back and confirming that this did work. I'm guessing that somewhere in the code there is a dependency on AWS sdk 1.1x.x? This effectively overwrites which version of the java SDK to use.
n
yeah that is what I thought, could find it on the github tho
no clue!
110 Views
Open in Slack
PreviousNext
Powered by