FAS
10/18/2024, 7:05 PMparticipant_stream table on s3 location: s3://mybucket/warehouse
where databasename:mydb
AWS Account#consumer:
FlinkSQL job reads from the table
CREATE CATALOG `mycatalog`
WITH (
'type'='iceberg',
'catalog-impl'='org.apache.iceberg.aws.glue.GlueCatalog',
'io-impl'='org.apache.iceberg.aws.s3.S3FileIO',
'warehouse'='s3://mybucket/warehouse'
);
USE CATALOG `mycatalog`;
CREATE TEMPORARY VIEW default_catalog.default_database.participant_distinct AS
SELECT *
FROM (SELECT *,
ROW_NUMBER() OVER (PARTITION BY
uuid
ORDER BY `timestamp`,`offset` desc) AS rownum
FROM mycatalog.mydb.participants_stream /*+ OPTIONS('streaming'='true', 'monitor-interval'='15s') */)
WHERE rownum = 1;
Question#1: Is this mycatalog a temporary flink pointer to the location where catalog is ?? Since i have cross account setup i have read Only access
*Question#2: Can i directly access the table name with database prefix after the use catalog or do i have to access it like mycatalog.my*db.participants_stream ?FAS
10/24/2024, 1:46 AMdefault database exist hence asked the upstream team to provide readOnly access to the database
Caused by: software.amazon.awssdk.services.glue.model.AccessDeniedException: User: arn:aws:sts::xxxxxxxxxx:assumed-role/mydb/aws-sdk-java-xxxxxxxxx is not authorized to perform: glue:GetDatabase on resource: arn:aws:glue:us-east-1:xxxxxxx:database/default because no identity-based policy allows the glue:GetDatabase action (Service: Glue, Status Code: 400, Request ID: xxxxxxxx)
1. Got following error
software.amazon.awssdk.core.exception.SdkClientException: Multiple HTTP implementations were found on the classpath. To avoid non-deterministic loading implementations, please explicitly provide an HTTP client via the client builders, set the software.amazon.awssdk.http.service.impl system property with the FQCN of the HTTP service to use as the default, or remove all but one HTTP implementation from the classpath. The multiple implementations found were: [software.amazon.awssdk.http.urlconnection.UrlConnectionSdkHttpService,software.amazon.awssdk.http.apache.ApacheSdkHttpService]
initially resolved it using System._setProperty_("software.amazon.awssdk.http.service.impl","software.amazon.awssdk.http.apache.ApacheSdkHttpService"); but it resurfaced with other stack trace and eventually excluded in my gradle build that resolved the issue
configurations.all {
exclude group: 'software.amazon.awssdk', module: 'http-url-connection'
}FAS
10/24/2024, 1:57 AMCREATE CATALOG `{{ ICEBERG_CATALOG | raw}}`
WITH (
'type'='iceberg',
'catalog-impl'='org.apache.iceberg.aws.glue.GlueCatalog',
'io-impl'='org.apache.iceberg.aws.s3.S3FileIO',
'warehouse'='{{ ICEBERG_URI |raw}}',
'glue.id' = 'aws-accountnumber',
'aws.region' = 'us-east-1'
);
It started using the role attached to the node on my EKS cluster in AWS Consumer Account. When i removed the glue.id it confused in accessing catalog but using the correct assumeRole that was bind with serviceAccount in EKS and AWS Producer account.
Eventually resolved this issue by adding following properties in the catalog
'client.assume-role.arn'= '[RoleARN]',
'client.credentials-provider'='software.amazon.awssdk.auth.credentials.WebIdentityTokenFileCredentialsProvider'
Seems like a bug as there is no documentation that states along with glue.id one has to specify the client.assume-role.arn