https://flink.apache.org/ logo
Join Slack
Powered by
Hi I have a single Dockerfile that includes PyFlin...
# troubleshooting
t
Hi I have a single Dockerfile that includes PyFlink code to deploy different pipelines. Two of the pipelines somehow require conflicting 
.jar
dependencies. Both pipelines work properly if I add the dependencies to the 
/opt/flink/lib
folder. As a solution I wanted to add in one of the pipelines and additional 
pipeline.jars
configuration which adds some more jars required for that specific pipeline. However, I keep on getting errors which indicate that the jars are not loaded correctly. I also tried to run the pipeline using 
./bin/flink run --python pipeline.py --jarfile jar_dependency.jar
, but that also did not work. Does anyone understand why this does not work? A potential workaround would be to to build different images for the different pipelines, but I rather not do that. Thanks in advance!
I have resolved the issue by doing some more research into why the dependencies were conflicting. Turns out that using the FileSystem connector (while having added the 
flink-sql-connector-hive-3.1.2
is added to the 
/lib
folder) gives the conflict. I got the following error:
Copy code
...
Caused by: java.lang.NoClassDefFoundError: org/apache/flink/table/planner/delegation/DialectFactory
...
The solution is to move the planner JAR as discussed in the documentation: https://nightlies.apache.org/flink/flink-docs-release-1.16/docs/connectors/table/hive/overview/
But I am still not sure why the solution with 
pipeline.jars
was not working.
m
If you use Hive, have you swapped the planners?
t
I have, and that did solve the original problem.
Thanks
5 Views
Open in Slack
PreviousNext
Powered by