Can anyone tell me what jars are required for `fli...
# troubleshooting
r
Can anyone tell me what jars are required for
flink:1.20+iceberg:1.8+nessie:0.103
? What version of hadoop is compatible with this setup?
e
I have the same problem, did you find anything? I'm using MinIO and I keep getting AWS errors with the Nessie catalog, unless I don't have it with Paimon and Delta.
r
haha yes sir, i'm about to save you a lot of time
Copy code
FROM flink:1.20.0

# this image will be optimized, individual run layers are in place for debugging purposes as of right now

WORKDIR /opt/flink/lib
RUN wget -q https://repo1.maven.org/maven2/org/apache/commons/commons-configuration2/2.11.0/commons-configuration2-2.11.0.jar
RUN wget -q https://repo1.maven.org/maven2/org/apache/commons/commons-lang3/3.17.0/commons-lang3-3.17.0.jar
RUN wget -q https://repo1.maven.org/maven2/org/apache/flink/flink-autoscaler-plugin-jdbc/1.10.0/flink-autoscaler-plugin-jdbc-1.10.0.jar
RUN wget -p https://repo1.maven.org/maven2/org/apache/flink/flink-autoscaler-standalone/1.10.0/flink-autoscaler-standalone-1.10.0.jar
RUN wget -q https://repo1.maven.org/maven2/org/apache/hadoop/hadoop-auth/3.4.1/hadoop-auth-3.4.1.jar
RUN wget -q https://repo1.maven.org/maven2/org/apache/hadoop/hadoop-aws/3.4.1/hadoop-aws-3.4.1.jar
RUN wget -q https://repo1.maven.org/maven2/org/apache/hadoop/hadoop-common/3.4.1/hadoop-common-3.4.1.jar
RUN wget -q https://repo1.maven.org/maven2/org/apache/hadoop/hadoop-client/3.4.1/hadoop-client-3.4.1.jar
RUN wget -q https://repo1.maven.org/maven2/org/apache/hadoop/hadoop-hdfs-client/3.4.1/hadoop-hdfs-client-3.4.1.jar
RUN wget -q https://repo1.maven.org/maven2/org/apache/hadoop/hadoop-mapreduce-client-core/3.4.1/hadoop-mapreduce-client-core-3.4.1.jar
RUN wget -q https://repo1.maven.org/maven2/org/apache/hadoop/thirdparty/hadoop-shaded-guava/1.3.0/hadoop-shaded-guava-1.3.0.jar
RUN wget -q https://repo1.maven.org/maven2/org/apache/httpcomponents/httpclient/4.5.9/httpclient-4.5.9.jar
RUN wget -q https://repo1.maven.org/maven2/org/apache/httpcomponents/httpcore/4.4.9/httpcore-4.4.9.jar
RUN wget -q https://repo1.maven.org/maven2/org/apache/iceberg/iceberg-flink-runtime-1.20/1.8.0/iceberg-flink-runtime-1.20-1.8.0.jar

RUN wget -q https://repo1.maven.org/maven2/commons-logging/commons-logging/1.3.5/commons-logging-1.3.5.jar
RUN wget -q https://repo1.maven.org/maven2/software/amazon/awssdk/apache-client/2.30.9/apache-client-2.30.9.jar
RUN wget -q https://repo1.maven.org/maven2/software/amazon/awssdk/auth/2.30.9/auth-2.30.9.jar
RUN wget -q https://repo1.maven.org/maven2/software/amazon/awssdk/aws-core/2.30.9/aws-core-2.30.9.jar
RUN wget -q https://repo1.maven.org/maven2/software/amazon/awssdk/aws-query-protocol/2.30.9/aws-query-protocol-2.30.9.jar
RUN wget -q https://repo1.maven.org/maven2/software/amazon/awssdk/aws-xml-protocol/2.30.9/aws-xml-protocol-2.30.9.jar
RUN wget -q https://repo1.maven.org/maven2/software/amazon/awssdk/checksums/2.30.9/checksums-2.30.9.jar
RUN wget -q https://repo1.maven.org/maven2/software/amazon/awssdk/checksums-spi/2.30.9/checksums-spi-2.30.9.jar
RUN wget -q https://repo1.maven.org/maven2/software/amazon/awssdk/endpoints-spi/2.30.9/endpoints-spi-2.30.9.jar
RUN wget -q https://repo1.maven.org/maven2/software/amazon/awssdk/http-auth/2.30.9/http-auth-2.30.9.jar
RUN wget -q https://repo1.maven.org/maven2/software/amazon/awssdk/http-auth-aws/2.30.9/http-auth-aws-2.30.9.jar
RUN wget -q https://repo1.maven.org/maven2/software/amazon/awssdk/http-auth-spi/2.30.9/http-auth-spi-2.30.9.jar
RUN wget -q https://repo1.maven.org/maven2/software/amazon/awssdk/http-client-spi/2.30.9/http-client-spi-2.30.9.jar
RUN wget -q https://repo1.maven.org/maven2/software/amazon/awssdk/identity-spi/2.30.9/identity-spi-2.30.9.jar
RUN wget -q https://repo1.maven.org/maven2/software/amazon/awssdk/json-utils/2.30.9/json-utils-2.30.9.jar
RUN wget -q https://repo1.maven.org/maven2/software/amazon/awssdk/metrics-spi/2.30.9/metrics-spi-2.30.9.jar
RUN wget -q https://repo1.maven.org/maven2/software/amazon/awssdk/profiles/2.30.9/profiles-2.30.9.jar
RUN wget -q https://repo1.maven.org/maven2/software/amazon/awssdk/protocol-core/2.30.9/protocol-core-2.30.9.jar
RUN wget -q https://repo1.maven.org/maven2/software/amazon/awssdk/regions/2.30.9/regions-2.30.9.jar
RUN wget -q https://repo1.maven.org/maven2/software/amazon/awssdk/retries/2.30.9/retries-2.30.9.jar
RUN wget -q https://repo1.maven.org/maven2/software/amazon/awssdk/retries-spi/2.30.9/retries-spi-2.30.9.jar
RUN wget -q https://repo1.maven.org/maven2/software/amazon/awssdk/s3/2.30.9/s3-2.30.9.jar
RUN wget -q https://repo1.maven.org/maven2/software/amazon/awssdk/s3-transfer-manager/2.30.9/s3-transfer-manager-2.30.9.jar
RUN wget -q https://repo1.maven.org/maven2/software/amazon/awssdk/sdk-core/2.30.9/sdk-core-2.30.9.jar
RUN wget -q https://repo1.maven.org/maven2/software/amazon/awssdk/sts/2.30.9/sts-2.30.9.jar
RUN wget -q https://repo1.maven.org/maven2/software/amazon/awssdk/third-party-jackson-core/2.30.9/third-party-jackson-core-2.30.9.jar
RUN wget -q https://repo1.maven.org/maven2/software/amazon/awssdk/utils/2.30.9/utils-2.30.9.jar

# woodstox, staxs, jackson and jaxb is neded for xml parsing, classes present in java8 were removed in java11
RUN wget -q https://repo1.maven.org/maven2/com/fasterxml/jackson/core/jackson-core/2.18.2/jackson-core-2.18.2.jar
RUN wget -q https://repo1.maven.org/maven2/com/fasterxml/jackson/core/jackson-databind/2.18.2/jackson-databind-2.18.2.jar
RUN wget -q https://repo1.maven.org/maven2/com/fasterxml/woodstox/woodstox-core/7.1.0/woodstox-core-7.1.0.jar
RUN wget -q https://repo1.maven.org/maven2/javax/xml/bind/jaxb-api/2.3.1/jaxb-api-2.3.1.jar
RUN wget -q https://repo1.maven.org/maven2/org/codehaus/woodstox/stax2-api/4.2.2/stax2-api-4.2.2.jar
RUN wget -q https://repo1.maven.org/maven2/org/glassfish/jaxb/jaxb-runtime/2.3.1/jaxb-runtime-2.3.1.jar
RUN wget -q https://repo1.maven.org/maven2/org/reactivestreams/reactive-streams/1.0.4/reactive-streams-1.0.4.jar


RUN mkdir /opt/flink/plugins/flink-s3-fs-presto
WORKDIR /opt/flink/plugins/flink-s3-fs-presto
RUN wget -q https://repo1.maven.org/maven2/org/apache/flink/flink-s3-fs-presto/1.20.0/flink-s3-fs-presto-1.20.0.jar

WORKDIR /opt/flink
ENTRYPOINT ["/docker-entrypoint.sh"]
i trimmed this down a little bit taking out other things i'm using like protobuf and kafka. this docker image should get you set up to run flink on aws with nessie as a catalog
The hadoop stuff is there because it is required for the sink table to take streaming data
πŸ™ 1