Does Flink support Amazon s3 Table buckets for Cat...
# troubleshooting
m
Does Flink support Amazon s3 Table buckets for Catalog definition? Similar to this example for Spark? I'm attempting to define a catalog on s3 table warehouse in Flink TableAPI, like this:
Copy code
tEnv.executeSql(
                    "CREATE CATALOG pulse_table WITH ("
                            + "'type'='iceberg',"
                            + "'warehouse'='arn:aws:s3tables:us-east-1:058264243434:bucket/kyk-dataeng-table-iceberg',"
                            + "'catalog-impl'='org.apache.iceberg.aws.s3tables.S3TablesCatalog',"
                            + ")"
            );
But that is not working for me. I have added:
Copy code
dependencies {
    implementation 'software.amazon.awssdk:s3tables:2.29.26'
    implementation 'software.amazon.s3tables:s3-tables-catalog-for-iceberg:0.1.7'
}
But I'm still getting:
Copy code
Exception in thread "main" java.lang.IllegalArgumentException: Cannot initialize Catalog implementation org.apache.iceberg.aws.s3tables.S3TablesCatalog: Cannot find constructor for interface org.apache.iceberg.catalog.Catalog
	Missing org.apache.iceberg.aws.s3tables.S3TablesCatalog [java.lang.ClassNotFoundException: org.apache.iceberg.aws.s3tables.S3TablesCatalog]
a
See this blog post, where rmoff describe this: https://apache-flink.slack.com/archives/C03FYR328H4/p1750860324250779
Oh, sorry. Not exactly the same. He's using the GlueCatalog. Not sure if this blog posts helps you
m
Yeah, this is what we're doing now, but I would like to use s3 Table bucket instead.
a
Looks like your using gradle? You'd either need to include the required dependencies inside a fat Jar file (see https://nightlies.apache.org/flink/flink-docs-release-1.20/docs/dev/configuration/gradle/#adding-dependencies-to-the-project), or bundle them with the Flink distribution inside the
lib
folder
I guess your also doing Flink SQL liek rmoff? Then I imagine you'd have to put the dependencies inside Flink's
lib
folder like rmoff does in this part of the blog: https://rmoff.net/2025/06/24/writing-to-apache-iceberg-on-s3-using-flink-sql-with-glue-catalog/#_jar_location
Oh. Again. I'm reading poorly today πŸ˜‰ I see that it looks like you're using the Java APIs. Then either option should work (fat jar and
lib
folder)
m
Thanks Andreas, I am building gradle locally and running this locally. I am aware of the need to push the dependencies to lib for the Flink SQL. But locally it is not working with the current inclusion of
Copy code
flinkShadowJar "software.amazon.awssdk:s3tables:2.29.26"
    flinkShadowJar "software.amazon.s3tables:s3-tables-catalog-for-iceberg:0.1.7"
And catalog definition as above.
Are you using s3 Table bucket catalog with flink via the TableAPI?
a
No we're not. So I don't know for sure that this should work. On potential issue: I'm guessing you should use
software.amazon.s3tables.iceberg.S3TablesCatalog
(not
org.apache.iceberg.aws.s3tables.S3TablesCatalog
) for the
catalog-impl
. I'm basing this on looking at the source for S3TablesCatalog. Have you tried that? If the above still fails, can you show the your full gradle file? Flink can be very finicky in terms of getting the classpath set up correctly. Also, if you use the config
flinkShadowJar
, then there shouldn't be a need for copying the dependencies to the
lib
folder, as they will get included in the jar file produced by gradle.
πŸ™ 1
m
Hi Andreas, thank you so much, that actually did fix the problem!
πŸ™Œ 1
a
You're welcome. Glad to hear it.