hei team! We are trying to setup a local environme...
# troubleshooting
o
hei team! We are trying to setup a local environment using flink+hive and we have been following this repository as reference: https://github.com/fhueske/flink-sql-demo/blob/master/docker-compose.yml We have had to change it a bit since we wanted to use Hive 3.1.3 and Hadoop 3.3.3. We are now facing an issue when submitting a job via flink UI. This is the stack trace: https://gist.github.com/mostrovoi/fcaeaf34c0cccb02ca2d17ce43ed5397 As you can see there seems to be a mismatch of versions and we suspect of the library. flink-shaded-hadoop2-uber . How can we know what version of shaded hadoop works with hive 3.1.0 ? We have tried to google for information but we could not find much about the purpose of this library or the aforementioned error. Thanks in advance!
m
The
flink-shaded-hadoop2.uber
isn't something that's actively being used/developed anymore
Hadoop dependencies should be loaded via
HADOOP_CLASSPATH
, as explained on https://nightlies.apache.org/flink/flink-docs-master/docs/dev/configuration/advanced/#hadoop-dependencies