Hi Everyone, I'd like to learn more about the nuan...
# troubleshooting
f
Hi Everyone, I'd like to learn more about the nuances of reading iceberg tables using AWS Glue in cross account and appreciate your feeback. AWS Account#producer: Producing iceberg tables using AWS Glue catalog on s3 in parquet format. Lets say
participant_stream
table on s3 location:
s3://mybucket/warehouse
where databasename:
mydb
AWS Account#consumer: FlinkSQL job reads from the table
Copy code
CREATE CATALOG  `mycatalog`
WITH (
    'type'='iceberg',
    'catalog-impl'='org.apache.iceberg.aws.glue.GlueCatalog',
    'io-impl'='org.apache.iceberg.aws.s3.S3FileIO',
    'warehouse'='s3://mybucket/warehouse'
);

USE CATALOG `mycatalog`;

CREATE TEMPORARY VIEW default_catalog.default_database.participant_distinct AS
SELECT *
FROM (SELECT *,
             ROW_NUMBER() OVER (PARTITION BY
                 uuid
                 ORDER BY `timestamp`,`offset` desc) AS rownum
      FROM mycatalog.mydb.participants_stream /*+  OPTIONS('streaming'='true', 'monitor-interval'='15s') */)
WHERE rownum = 1;
Question#1: Is this mycatalog a temporary flink pointer to the location where catalog is ?? Since i have cross account setup i have read Only access *Question#2: Can i directly access the table name with database prefix after the use catalog or do i have to access it like
mycatalog.my*db.participants_stream
?
There are few interesting issues identified while getting this solution work 1. Even though we have database prefixed, but GlueCatalog tried to check if
default
database exist hence asked the upstream team to provide readOnly access to the database
Copy code
Caused by: software.amazon.awssdk.services.glue.model.AccessDeniedException: User: arn:aws:sts::xxxxxxxxxx:assumed-role/mydb/aws-sdk-java-xxxxxxxxx is not authorized to perform: glue:GetDatabase on resource: arn:aws:glue:us-east-1:xxxxxxx:database/default because no identity-based policy allows the glue:GetDatabase action (Service: Glue, Status Code: 400, Request ID: xxxxxxxx)
1. Got following error
Copy code
software.amazon.awssdk.core.exception.SdkClientException: Multiple HTTP implementations were found on the classpath. To avoid non-deterministic loading implementations, please explicitly provide an HTTP client via the client builders, set the software.amazon.awssdk.http.service.impl system property with the FQCN of the HTTP service to use as the default, or remove all but one HTTP implementation from the classpath.  The multiple implementations found were: [software.amazon.awssdk.http.urlconnection.UrlConnectionSdkHttpService,software.amazon.awssdk.http.apache.ApacheSdkHttpService]
initially resolved it using
System._setProperty_("software.amazon.awssdk.http.service.impl","software.amazon.awssdk.http.apache.ApacheSdkHttpService");
but it resurfaced with other stack trace and eventually excluded in my gradle build that resolved the issue
Copy code
configurations.all {
    exclude group: 'software.amazon.awssdk', module: 'http-url-connection'
}
Issue#3: When i used glue.id as follows
Copy code
CREATE CATALOG  `{{ ICEBERG_CATALOG | raw}}`
WITH (
    'type'='iceberg',
    'catalog-impl'='org.apache.iceberg.aws.glue.GlueCatalog',
    'io-impl'='org.apache.iceberg.aws.s3.S3FileIO',
    'warehouse'='{{ ICEBERG_URI |raw}}',
    'glue.id' = 'aws-accountnumber',
    'aws.region' = 'us-east-1'
);
It started using the role attached to the node on my EKS cluster in AWS Consumer Account. When i removed the
glue.id
it confused in accessing catalog but using the correct assumeRole that was bind with serviceAccount in EKS and AWS Producer account. Eventually resolved this issue by adding following properties in the catalog
Copy code
'client.assume-role.arn'= '[RoleARN]',    
'client.credentials-provider'='software.amazon.awssdk.auth.credentials.WebIdentityTokenFileCredentialsProvider'
Seems like a bug as there is no documentation that states along with
glue.id
one has to specify the
client.assume-role.arn