https://flink.apache.org/ logo
Join Slack
Powered by
What's the right way to get around the following e...
# troubleshooting
n
What's the right way to get around the following error when writing a unit test for pyflink?
Copy code
Could not found the Java class 'org.apache.flink.runtime.testutils.MiniClusterResourceConfiguration.Builder'. The Java dependencies could be specified via command line argument '--jarfile' or the config option 'pipeline.jars'
Slight progress that I was able to download 
flink-runtime-1.16.1-tests.jar
and place it in the discovery path of starting up the gateway and got met with different errors
Copy code
py4j.protocol.Py4JError: org.apache.flink.test.util.MiniClusterWithClientResource does not exist in the JVM
That appears to want 
flink-test-utils-1.16.1-sources.jar
since I only see that there. Adding that to the path doesn't help.
Copy code
$ jar tf venv/lib/python3.8/flink-python/target/test-dependencies/flink-test-utils-1.16.1-sources.jar | grep MiniClusterWithClientResource
org/apache/flink/test/util/MiniClusterWithClientResource.java
Very noob to java, but I suspect I'm just not gathering all the jar dependencies. Going to try to grab them with maven
Okay, I figured it out in case anyone was interested. I'm going to look for a more permanent solution, but here's the minimum jars needed to add to the class path when starting the jvm
Copy code
flink-runtime-1.16.1-tests.jar
flink-test-utils-1.16.1.jar
hamcrest-core-1.3.jar
junit-4.13.2.jar
You have to put them somewhere 
pyflink_gateway_server.py:construct_test_classpath
is looking. In our repo, I'll likely override that to include a custom location rather than what is included from the source. With all of these things in place, the unit testing library can be used to set up a simple mini cluster.
I've opened https://issues.apache.org/jira/browse/FLINK-31073 in case others come across this later.
d
@Nathanael England Thanks a lot for the sharing!
πŸ‘ 1
Open in Slack
PreviousNext
Powered by