Channels
acryl-omnisend
advice-data-governance
advice-metadata-modeling
all-things-datahub-in-windows
all-things-deployment
announcements
authentication-authorization
chatter
column-level-lineage
contribute
contribute-datahub-blog
datahub-soda-test
demo-slack-notifications
design-business-glossary
design-data-product-entity
design-data-quality
design-datahub-documentation
design-dataset-access-requests
design-dataset-joins
feature-requests
flyte-datahub-integration
getting-started
github-activities
help-
i18n-community-contribution
ingestion
integration-alteryx-datahub
integration-azure-datahub
integration-dagster-datahub
integration-databricks-datahub
integration-datastudio-datahub
integration-iceberg-datahub
integration-powerbi-datahub
integration-prefect-datahub
integration-protobuf
integration-tableau-datahub
integration-vertica-datahub
introduce-yourself
jobs
metadata-day22-hackathon
muti-tenant-deployment
office-hours
openapi
plugins
show-and-tell
talk-data-product-management
troubleshoot
ui
Title
g
gorgeous-dinner-4055
01/27/2022, 5:00 PMHello All!
We've got most things in our deployment sorted out, but stuck on one last bug with fetching AWS credentials. I know a couple other people have deployed using mostly managed AWS services, so wanted to see if anyone else has run into the same issues as us and how you solved it.
What we're seeing:
Calls to AWS Glue to fetch schemas seems to fail with credentials not found(See log in thread).
However, if I ssh into the pod, I am able to run glue fetch schemas, get current role, etc with the CLI. So the role seems valid, and the latest version of AWS cli seems to pick up the role just fine. The Glue and AWS dependency seem pretty up to date. Any idea on what else could be causing this issue?
We've seen this type of issue before when the AWS library is out of data, and our workaround was to have a background thread that refreshes credentials to a credential file. So we can use that same solution, but ofc would prefer not to.
Caused by: software.amazon.awssdk.core.exception.SdkClientException: Unable to load credentials from any of the providers in the chain AwsCredentialsProviderChain(credentialsProviders=[SystemPropertyCredentialsProvider(), EnvironmentVariableCredentialsProvider(), WebIdentityTokenCredentialsProvider(), ProfileCredentialsProvider(), ContainerCredentialsProvider(), InstanceProfileCredentialsProvider()]) : [SystemPropertyCredentialsProvider(): Unable to load credentials from system settings. Access key must be specified either via environment variable (AWS_ACCESS_KEY_ID) or system property (aws.accessKeyId)., EnvironmentVariableCredentialsProvider(): Unable to load credentials from system settings. Access key must be specified either via environment variable (AWS_ACCESS_KEY_ID) or system property (aws.accessKeyId)., WebIdentityTokenCredentialsProvider(): Multiple HTTP implementations were found on the classpath. To avoid non-deterministic loading implementations, please explicitly provide an HTTP client via the client builders, set the software.amazon.awssdk.http.service.impl system property with the FQCN of the HTTP service to use as the default, or remove all but one HTTP implementation from the classpath, ProfileCredentialsProvider(): Profile file contained no credentials for profile 'default': ProfileFile(profiles=[]), ContainerCredentialsProvider(): Cannot fetch credentials from container - neither AWS_CONTAINER_CREDENTIALS_FULL_URI or AWS_CONTAINER_CREDENTIALS_RELATIVE_URI environment variables are set., InstanceProfileCredentialsProvider(): The requested metadata is not found at <http://169.254.169.254/latest/meta-data/iam/security-credentials/>]From reading the AWS docs the DefaultCredentialProvider should be able to pickup the IAM role
b
big-carpet-38439
01/27/2022, 7:46 PMAh this is instead of providing access key / secret as variables inside of the recipe?
g
gorgeous-dinner-4055
01/27/2022, 7:48 PMI'm unfamiliar with what recipe means, but this occurs when the datahub-gms service tries to ingest anything.
But yes, if a key/secret isn't in the env variables for datahub-gms, ingestion fails. But if we set temp creds(secret_key, secret_id, token), ingestion starts working
b
big-carpet-38439
01/27/2022, 7:54 PMRecipe just meaning the YML configuration for your ingestion!
g
gorgeous-dinner-4055
01/27/2022, 8:01 PMAh gotcha, thanks for the clarification!
So there's no Recipe involved. We are using the Python REST API to emit dataset aspects. The failure is when the gms service is ingesting after receiving a request.
The failure looks to be coming from this code: https://github.com/linkedin/datahub/blob/27a5f6dc3b2339e853a0d2dd0956a6a1bb8a10c5/[…]/factory/kafka/schemaregistry/AwsGlueSchemaRegistryFactory.java
but nothing in that code is suspicious in any way. Here's the stack trace if it's useful:
at software.amazon.awssdk.core.exception.SdkClientException$BuilderImpl.build(SdkClientException.java:98)
at software.amazon.awssdk.auth.credentials.AwsCredentialsProviderChain.resolveCredentials(AwsCredentialsProviderChain.java:112)
at software.amazon.awssdk.auth.credentials.internal.LazyAwsCredentialsProvider.resolveCredentials(LazyAwsCredentialsProvider.java:45)
at software.amazon.awssdk.auth.credentials.DefaultCredentialsProvider.resolveCredentials(DefaultCredentialsProvider.java:104)
at software.amazon.awssdk.awscore.client.handler.AwsClientHandlerUtils.createExecutionContext(AwsClientHandlerUtils.java:79)
at software.amazon.awssdk.awscore.client.handler.AwsSyncClientHandler.createExecutionContext(AwsSyncClientHandler.java:68)
at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.lambda$execute$1(BaseSyncClientHandler.java:99)
at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.measureApiCallSuccess(BaseSyncClientHandler.java:169)
at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.execute(BaseSyncClientHandler.java:95)
at software.amazon.awssdk.core.client.handler.SdkSyncClientHandler.execute(SdkSyncClientHandler.java:45)
at software.amazon.awssdk.awscore.client.handler.AwsSyncClientHandler.execute(AwsSyncClientHandler.java:55)
at software.amazon.awssdk.services.glue.DefaultGlueClient.getSchemaByDefinition(DefaultGlueClient.java:6837)
at com.amazonaws.services.schemaregistry.common.AWSSchemaRegistryClient.getSchemaVersionIdByDefinition(AWSSchemaRegistryClient.java:134)b
big-carpet-38439
01/28/2022, 12:21 AMOh i see - thank you!
If you log into that container, and attempt to echo the
AWS_ACCESS_KEY_IDg
gorgeous-dinner-4055
01/28/2022, 12:54 AMAWS_ACCESS_KEY_IDbash-5.1$ env | grep AWS
AWS_DEFAULT_REGION=xxxxxx
AWS_REGION=xxxxxxx
AWS_ROLE_ARN=arn:aws:iam::00000000:role/xxxxxx
AWS_WEB_IDENTITY_TOKEN_FILE=/xxxxxxxxn
numerous-camera-74294
02/18/2022, 6:38 PMhi! did you manage to solve this? I am facing this exact error
Caused by: software.amazon.awssdk.core.exception.SdkClientException: Unable to load credentials from any of the providers in the chain AwsCredentialsProviderChain(credentialsProviders=[SystemPropertyCredentialsProvider(), EnvironmentVariableCredentialsProvider(), WebIdentityTokenCredentialsProvider(), ProfileCredentialsProvider(), ContainerCredentialsProvider(), InstanceProfileCredentialsProvider()]) : [SystemPropertyCredentialsProvider(): Unable to load credentials from system settings. Access key must be specified either via environment variable (AWS_ACCESS_KEY_ID) or system property (aws.accessKeyId)., EnvironmentVariableCredentialsProvider(): Unable to load credentials from system settings. Access key must be specified either via environment variable (AWS_ACCESS_KEY_ID) or system property (aws.accessKeyId)., WebIdentityTokenCredentialsProvider(): Either the environment variable AWS_WEB_IDENTITY_TOKEN_FILE or the javaproperty aws.webIdentityTokenFile must be set., ProfileCredentialsProvider(): Profile file contained no credentials for profile 'default': ProfileFile(profiles=[]), ContainerCredentialsProvider(): Cannot fetch credentials from container - neither AWS_CONTAINER_CREDENTIALS_FULL_URI or AWS_CONTAINER_CREDENTIALS_RELATIVE_URI environment variables are set., InstanceProfileCredentialsProvider(): Unable to load credentials from service endpoint.]
at software.amazon.awssdk.core.exception.SdkClientException$BuilderImpl.build(SdkClientException.java:98)
at software.amazon.awssdk.auth.credentials.AwsCredentialsProviderChain.resolveCredentials(AwsCredentialsProviderChain.java:112)
at software.amazon.awssdk.auth.credentials.internal.LazyAwsCredentialsProvider.resolveCredentials(LazyAwsCredentialsProvider.java:45)
at software.amazon.awssdk.auth.credentials.DefaultCredentialsProvider.resolveCredentials(DefaultCredentialsProvider.java:104)
at software.amazon.awssdk.awscore.client.handler.AwsClientHandlerUtils.createExecutionContext(AwsClientHandlerUtils.java:79)
at software.amazon.awssdk.awscore.client.handler.AwsSyncClientHandler.createExecutionContext(AwsSyncClientHandler.java:68)
at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.lambda$execute$1(BaseSyncClientHandler.java:99)
at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.measureApiCallSuccess(BaseSyncClientHandler.java:169)
at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.execute(BaseSyncClientHandler.java:95)
at software.amazon.awssdk.core.client.handler.SdkSyncClientHandler.execute(SdkSyncClientHandler.java:45)
at software.amazon.awssdk.awscore.client.handler.AwsSyncClientHandler.execute(AwsSyncClientHandler.java:55)
at software.amazon.awssdk.services.glue.DefaultGlueClient.getSchemaByDefinition(DefaultGlueClient.java:6837)
at com.amazonaws.services.schemaregistry.common.AWSSchemaRegistryClient.getSchemaVersionIdByDefinition(AWSSchemaRegistryClient.java:134)
... 50 common frames omittedg
gorgeous-dinner-4055
02/18/2022, 11:21 PMI could not figure out what was the issue here. Here were the things i tried to see if an alternative solution worked:
1. Wrote to ~/.aws credentials with temporary credentials. I was hoping to maybe run a background thread to refresh creds at some cadends. That didn't work
2. Bumped up the version of Glue dependency to see if there was an issue with the AWS package being out of date - Didn't work
The ROI for this was starting to be small, so I ended up creating a user and adding in the static creds of the user into the environment variables on startup(Awful, I know 😞 )
When I logged into the pod, I was able to call Glue using the AWS cli and inspect the schema, so clearly the role was working for the pod, but for some reason the glue-schema-regisry was not working
and in the stack trace you can clearly see that
WebIdentityTokenCredentialsProvidern
numerous-camera-74294
02/20/2022, 9:48 AMthank you @gorgeous-dinner-4055, would be awesome if someone could look into this
@little-megabyte-1074 can you help us please?
w
wonderful-jordan-36532
02/21/2022, 12:26 PMWould be great if the aws credentials could be read from the kuberentes pod. Would provide an alternative to hard coding the credentials in recipe
n
numerous-camera-74294
02/22/2022, 3:48 PM@gorgeous-dinner-4055 just for you to know
I was able to bypass this by setting explicitly the implementation than I want to use
something like
extraEnvs:
- name: JAVA_OPTS
value: "-Dsoftware.amazon.awssdk.http.service.impl=software.amazon.awssdk.http.urlconnection.UrlConnectionSdkHttpService"in the values file of the datahub-gms
did the trick
g
gorgeous-dinner-4055
02/22/2022, 4:47 PMOh, this is awesome, thanks so much @numerous-camera-74294!!! I'll give that a try!
Just circling back and confirming that this did work.
I'm guessing that somewhere in the code there is a dependency on AWS sdk 1.1x.x? This effectively overwrites which version of the java SDK to use.
n
numerous-camera-74294
02/23/2022, 3:27 PMyeah that is what I thought, could find it on the github tho
no clue!