https://flink.apache.org/ logo
Join Slack
Powered by
Hello everyone, I recently updated flink to 1.17.1...
# troubleshooting
d
Hello everyone, I recently updated flink to 1.17.1 and I started to get this error .
Copy code
2023-06-15 18:41:30,374 INFO  org.apache.flink.runtime.security.token.DelegationTokenReceiverRepository [] - Delegation token receiver s3-hadoop loaded and initialized
2023-06-15 18:41:30,375 ERROR org.apache.flink.runtime.security.token.DelegationTokenReceiverRepository [] - Failed to initialize delegation token receiver s3-hadoop
java.lang.IllegalStateException: Delegation token receiver with service name {} has multiple implementations [s3-hadoop]
        at org.apache.flink.util.Preconditions.checkState(Preconditions.java:215) ~[flink-dist-1.17.1.jar:1.17.1]
        at org.apache.flink.runtime.security.token.DelegationTokenReceiverRepository.lambda$loadReceivers$0(DelegationTokenReceiverRepository.java:75) ~[flink-dist-1.17.1.jar:1.17.1]
        at java.util.Iterator.forEachRemaining(Iterator.java:116) ~[?:1.8.0_372]
        at org.apache.flink.runtime.security.token.DelegationTokenReceiverRepository.loadReceivers(DelegationTokenReceiverRepository.java:98) ~[flink-dist-1.17.1.jar:1.17.1]
        at org.apache.flink.runtime.security.token.DelegationTokenReceiverRepository.<init>(DelegationTokenReceiverRepository.java:60) ~[flink-dist-1.17.1.jar:1.17.1]
        at org.apache.flink.runtime.taskexecutor.TaskManagerRunner.startTaskManagerRunnerServices(TaskManagerRunner.java:245) ~[flink-dist-1.17.1.jar:1.17.1]
        at org.apache.flink.runtime.taskexecutor.TaskManagerRunner.start(TaskManagerRunner.java:293) ~[flink-dist-1.17.1.jar:1.17.1]
        at org.apache.flink.runtime.taskexecutor.TaskManagerRunner.runTaskManager(TaskManagerRunner.java:486) ~[flink-dist-1.17.1.jar:1.17.1]
        at org.apache.flink.runtime.taskexecutor.TaskManagerRunner.lambda$runTaskManagerProcessSecurely$5(TaskManagerRunner.java:530) ~[flink-dist-1.17.1.jar:1.17.1]
        at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_372]
        at javax.security.auth.Subject.doAs(Subject.java:422) [?:1.8.0_372]
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1899) [hadoop-common-3.3.5.jar:?]
        at org.apache.flink.runtime.security.contexts.HadoopSecurityContext.runSecured(HadoopSecurityContext.java:41) [flink-dist-1.17.1.jar:1.17.1]
        at org.apache.flink.runtime.taskexecutor.TaskManagerRunner.runTaskManagerProcessSecurely(TaskManagerRunner.java:530) [flink-dist-1.17.1.jar:1.17.1]
        at org.apache.flink.runtime.taskexecutor.TaskManagerRunner.runTaskManagerProcessSecurely(TaskManagerRunner.java:510) [flink-dist-1.17.1.jar:1.17.1]
        at org.apache.flink.runtime.taskexecutor.TaskManagerRunner.main(TaskManagerRunner.java:468) [flink-dist-1.17.1.jar:1.17.1]
my hadoop version is 3.3.5, hive 3.1.3, scala 2.12, iceburg 1.3.0 and serval plugins S3, azure blob, AWS_SDK_VERSION 2.20.18. and for local dev we are using minio for s3. (edit) adding security.delegation.token.provider.s3-hadoop.enabled: false para to flinkconf.yaml has solved to issue of none start, but I would like a better solution without having to disable anything.
s
this error is fixed as part of https://issues.apache.org/jira/browse/FLINK-31839 can you check if your PATH contains conflicting classes
d
Nothing i can spot on first glance maybe iam missing something
Copy code
FROM flink:1.17.1-scala_2.12-java11
#wget <https://archive.apache.org/dist/flink/flink-${FLINK_VERSION}/flink-${FLINK_VERSION}-bin-scala_${SCALA_VERSION}.tgz>

ENV HADOOP_VERSION=3.3.5
ENV JDBC_POSTQRES_VERSION=42.6.0
ENV HIVE_VERSION=3.1.3
ENV FLINK_VERSION_MAJOR=1.17
ENV FLINK_VERSION=${FLINK_VERSION_MAJOR}.1
ENV SCALA_VERSION=2.12
ENV ICEBERG_VERSION=1.3.0
ENV MAVEN_URL=<https://repo1.maven.org/maven2>
ENV ICEBERG_MAVEN_URL=${MAVEN_URL}/org/apache/iceberg
ENV APACHE_FLINK_URL=<https://archive.apache.org/dist/flink/>
ENV HADOOP_HOME /opt/hadoop-$HADOOP_VERSION 
#ENV HIVE_HOME=/opt/metastore

RUN apt-get update -y

# To build Flink image on bare OS
#RUN wget ${APACHE_FLINK_URL}/flink-${FLINK_VERSION}/flink-${FLINK_VERSION}-bin-scala_${SCALA_VERSION}.tgz && \
#    tar xzvf flink-${FLINK_VERSION}-bin-scala_${SCALA_VERSION}.tgz

# Install Hadoop
RUN cd /opt && \
    wget <https://archive.apache.org/dist/hadoop/common/hadoop-${HADOOP_VERSION}/hadoop-${HADOOP_VERSION}.tar.gz> && \
    tar xfz hadoop-${HADOOP_VERSION}.tar.gz  && \
    rm hadoop-${HADOOP_VERSION}.tar.gz


#ENV HADOOP_CLASSPATH=$(/opt/hadoop-3.3.5/bin/hadoop classpath) # known issue <https://github.com/docker/docker/issues/29110>
ENV HADOOP_CLASSPATH=${HADOOP_CLASSPATH}:${HADOOP_HOME}/share/hadoop/common/*
ENV HADOOP_CLASSPATH=${HADOOP_CLASSPATH}:${HADOOP_HOME}/share/hadoop/common/lib/*
ENV HADOOP_CLASSPATH=${HADOOP_CLASSPATH}:${HADOOP_HOME}/share/hadoop/hdfs/*
ENV HADOOP_CLASSPATH=${HADOOP_CLASSPATH}:${HADOOP_HOME}/share/hadoop/hdfs/lib/*
ENV HADOOP_CLASSPATH=${HADOOP_CLASSPATH}:${HADOOP_HOME}/share/hadoop/tools/lib/*
ENV HADOOP_CLASSPATH=${HADOOP_CLASSPATH}:${HADOOP_HOME}/share/hadoop/yarn/*
ENV HADOOP_CLASSPATH=${HADOOP_CLASSPATH}:${HADOOP_HOME}/share/hadoop/yarn/lib/*
ENV HADOOP_CLASSPATH=${HADOOP_CLASSPATH}:${HADOOP_HOME}/share/hadoop/mapreduce/*
ENV HADOOP_CONF_DIR=/opt/hadoop/conf

# HIVE dependencies
RUN cd ${FLINK_HOME}/lib && \
    wget ${MAVEN_URL}/org/apache/flink/flink-sql-connector-hive-${HIVE_VERSION}_${SCALA_VERSION}/${FLINK_VERSION}/flink-sql-connector-hive-${HIVE_VERSION}_${SCALA_VERSION}-${FLINK_VERSION}.jar && \
    wget ${MAVEN_URL}/org/apache/flink/flink-connector-hive_${SCALA_VERSION}/${FLINK_VERSION}/flink-connector-hive_${SCALA_VERSION}-${FLINK_VERSION}.jar && \
    wget ${MAVEN_URL}/org/apache/hive/hive-exec/${HIVE_VERSION}/hive-exec-${HIVE_VERSION}.jar && \
    wget ${MAVEN_URL}/org/apache/thrift/libfb303/0.9.3/libfb303-0.9.3.jar && \
    wget ${MAVEN_URL}/org/antlr/antlr-runtime/3.5.2/antlr-runtime-3.5.2.jar


RUN cd ${FLINK_HOME}/lib && \
    wget ${ICEBERG_MAVEN_URL}/iceberg-flink-runtime-${FLINK_VERSION_MAJOR}/${ICEBERG_VERSION}/iceberg-flink-runtime-${FLINK_VERSION_MAJOR}-${ICEBERG_VERSION}.jar && \
    wget  ${ICEBERG_MAVEN_URL}/iceberg-aws/${ICEBERG_VERSION}/iceberg-aws-${ICEBERG_VERSION}.jar

#RUN cd ${FLINK_HOME}/lib && \
#    wget ${MAVEN_URL}/org/apache/flink/flink-avro/${FLINK_VERSION}/flink-avro-${FLINK_VERSION}.jar

RUN mv ${FLINK_HOME}/opt/flink-table-planner_${SCALA_VERSION}-${FLINK_VERSION}.jar ${FLINK_HOME}/lib/flink-table-planner_${SCALA_VERSION}-${FLINK_VERSION}.jar && \
    mv ${FLINK_HOME}/lib/flink-table-planner-loader-${FLINK_VERSION}.jar ${FLINK_HOME}/opt/flink-table-planner-loader-${FLINK_VERSION}.jar


 ENV AWS_SDK_VERSION=2.20.18
 ENV AWS_MAVEN_URL=$MAVEN_URL/software/amazon/awssdk
 RUN cd ${FLINK_HOME}/lib && \
     wget ${AWS_MAVEN_URL}/bundle/${AWS_SDK_VERSION}/bundle-${AWS_SDK_VERSION}.jar && \
     wget ${AWS_MAVEN_URL}/url-connection-client/${AWS_SDK_VERSION}/url-connection-client-${AWS_SDK_VERSION}.jar

# s3 plugin
RUN cd ${FLINK_HOME}/plugins && \
    mkdir s3 &&  \
    cp ${FLINK_HOME}/opt/flink-s3-fs-hadoop-*.jar ${FLINK_HOME}/plugins/s3/

# azure blob plugin
 RUN cd ${FLINK_HOME}/plugins && \
     mkdir azure &&  \
     cp ${FLINK_HOME}/opt/flink-azure-fs-hadoop-*.jar ${FLINK_HOME}/plugins/azure/

RUN cd ${FLINK_HOME}/lib && \
    wget ${MAVEN_URL}/org/apache/flink/flink-json/${FLINK_VERSION}/flink-json-${FLINK_VERSION}.jar && \
    wget ${MAVEN_URL}/org/apache/flink/flink-sql-avro-confluent-registry/${FLINK_VERSION}/flink-sql-avro-confluent-registry-${FLINK_VERSION}.jar && \
    wget ${MAVEN_URL}/org/apache/flink/flink-connector-jdbc/3.1.0-1.17/flink-connector-jdbc-3.1.0-1.17.jar && \
    wget ${MAVEN_URL}/org/apache/flink/flink-sql-connector-kafka/${FLINK_VERSION}/flink-sql-connector-kafka-${FLINK_VERSION}.jar && \
#    jdbc drives
    wget ${MAVEN_URL}/mysql/mysql-connector-java/8.0.28/mysql-connector-java-8.0.28.jar && \
    wget ${MAVEN_URL}/org/mongodb/mongo-java-driver/3.12.8/mongo-java-driver-3.12.8.jar && \
    wget <https://jdbc.postgresql.org/download/postgresql-${JDBC_POSTQRES_VERSION}.jar> && \
#    jdbc cdc
    wget ${MAVEN_URL}/com/ververica/flink-connector-debezium/2.3.0/flink-connector-debezium-2.3.0.jar && \
#    wget ${MAVEN_URL}/com/ververica/flink-format-changelog-json/2.1.1/flink-format-changelog-json-2.1.1.jar &&\
    wget ${MAVEN_URL}/com/ververica/flink-connector-mysql-cdc/2.3.0/flink-connector-mysql-cdc-2.3.0.jar && \
    wget ${MAVEN_URL}/com/ververica/flink-connector-postgres-cdc/2.3.0/flink-connector-postgres-cdc-2.3.0.jar && \
#    wget ${MAVEN_URL}/com/ververica/flink-cdc-base/2.3.0/flink-cdc-base-2.3.0.jar && \
    wget ${MAVEN_URL}/org/apache/flink/flink-connector-files/${FLINK_VERSION}/flink-connector-files-${FLINK_VERSION}.jar && \
#    debezium dependencies
    wget ${MAVEN_URL}/io/debezium/debezium-core/2.1.0.Final/debezium-core-2.1.0.Final.jar && \
    wget ${MAVEN_URL}/io/debezium/debezium-api/2.1.0.Final/debezium-api-2.1.0.Final.jar && \
    wget ${MAVEN_URL}/io/debezium/debezium-embedded/2.1.0.Final/debezium-embedded-2.1.0.Final.jar && \
    wget <https://repo1.maven.org/maven2/io/debezium/debezium-connector-postgres/2.1.0.Final/debezium-connector-postgres-2.1.0.Final.jar> && \
#    kafka connector
    wget ${MAVEN_URL}/org/apache/kafka/connect/3.3.0/connect-3.3.0.jar && \
    wget ${MAVEN_URL}/org/apache/kafka/connect-api/3.3.0/connect-api-3.3.0.jar

#    flink table store connector
# RUN cd ${FLINK_HOME}/lib && \
#     wget <https://repository.apache.org/content/groups/snapshots/org/apache/paimon/paimon-flink-1.16/0.5-SNAPSHOT/paimon-flink-1.16-0.5-20230511.001935-7.jar> && \
#     wget <https://repository.apache.org/content/groups/snapshots/org/apache/paimon/paimon-hive-connector-3.1/0.5-SNAPSHOT/paimon-hive-connector-3.1-0.5-20230511.001935-6.jar>
#    wget <https://www.apache.org/dyn/closer.lua/flink/flink-table-store-0.3.0/flink-table-store-dist-0.3.0.jar> && \
#    wget <https://repo1.maven.org/maven2/org/apache/flink/flink-table-store-hive-catalog/0.3.0/flink-table-store-hive-catalog-0.3.0.jar> && \
#    wget <https://www.apache.org/dyn/closer.lua/flink/flink-table-store-0.3.0/flink-table-store-hive-catalog-0.3.0_3.1.jar>


#<https://repo1.maven.org/maven2/org/apache/orc/orc-core/1.8.3/orc-core-1.8.3.jar>
#<https://repo1.maven.org/maven2/io/airlift/aircompressor/0.24/aircompressor-0.24.jar>
# Orc dependencies -- required by the ORC vectorized optimizations
#RUN cd ${FLINK_HOME}/lib/ && \
#    mkdir hadoop-shaded &&  \
#    cd hadoop-shaded && \
#    wget <https://repository.cloudera.com/artifactory/cloudera-repos/org/apache/flink/flink-shaded-hadoop-3-uber/3.1.1.7.2.9.0-173-9.0/flink-shaded-hadoop-3-uber-3.1.1.7.2.9.0-173-9.0.jar>
#ENV ${CLASSPATH};CLASSPATH=${FLINK_HOME}/lib/hadoop-shaded

ENV CLASSPATH=${HADOOP_CLASSPATH}:${CLASSPATH}
#COPY ./platform-conf/hive-site.xml /opt/hive-conf/
s
let me try to reproduce this error and find the root cause for it .
can you share repro steps if you have handy ?
m
<https://repository.cloudera.com/artifactory/cloudera-repos/org/apache/flink/flink-shaded[…]-9.0/flink-shaded-hadoop-3-uber-3.1.1.7.2.9.0-173-9.0.jar>
I have sincere doubts about this one: there's no Flink Hadoop shaded JAR, this is from a 3rd party vendor. You shouldn't use any Flink Shaded Hadoop/Uber JAR
I think you're basically in a dependency hell: there are all sorts of different Hadoop versions being pulled in, with loads of transitive dependencies that are different too. I would recommend to first scope down your setup, and then add one at a time to see where it goes wrong
That would be weird, given that those or one of the few that are self contained
Ah your message is gone 😂
d
yeah one sec
i was trying to get code block to work
but anyway, i fixed it by getting rid of the commands that add s3 plugin to plugins file, it was causing a conflict, because it seems that flink added s3 plugin by default, and adding it again cause the multiple implicitization error. although documentations still says to add the plugin in the image.
m
Flink doesn’t add S3 plugins by default
d
weird then, I dont know whats doing it. but its getting added twice.
i removed this code and it still adds it so, iam a little confused but it works now and still adds it to plugins folder. Thanks you for replying
Copy code
mkdir ./plugins/s3-fs-hadoop
cp ./opt/flink-s3-fs-hadoop-1.17.1.jar ./plugins/s3-fs-hadoop/
m
Yes you are copying it yourself from opt to plugins
Happy to hear it’s resolved now
2 Views
Open in Slack
PreviousNext
Powered by