When running a LOT of PyFlink queries (giant unit ...
# troubleshooting
d
When running a LOT of PyFlink queries (giant unit test suite, let's say this starts happening a few hundred queries in) using Flink mini-cluster, I'm eventually running into:
Copy code
>                   raise Py4JJavaError(
                        "An error occurred while calling {0}{1}{2}.\n".
                        format(target_id, ".", name), value)
E                   py4j.protocol.Py4JJavaError: An error occurred while calling o8.sqlQuery.
E                   : java.lang.OutOfMemoryError: Compressed class space
I'm not sure what the fix is; something I found is https://lists.apache.org/thread/6ls2m9oslj5w9l9rojfv22nskp635hmy. Maybe https://stackoverflow.com/questions/62879157/usecompressedoops-usecompressedclasspointers-in-jdk-13-and-jdk-15 could be something else to look into. However, I'm not sure what I'm doing is effective. I've tried, when creating the table environment with PyFlink, all sorts of combinations of:
Copy code
table_env.get_config().set("env.java.opts.all", "-XX:CompressedClassSpaceSize=100M -XX:MaxMetaspaceSize=200M")  # <https://lists.apache.org/thread/6ls2m9oslj5w9l9rojfv22nskp635hmy>
as well as
Copy code
table_env.get_config().set("parallelism.default", "1")
but things still fail. I also tried to debug with:
Copy code
table_env.get_config().set("env.java.opts.all", "-XX:+PrintFlagsFinal")
        table_env.get_config().set("env.java.opts.all", "-XX:NativeMemoryTracking=summary")
        table_env.get_config().set("env.java.opts.client", "-XX:+PrintFlagsFinal")
        table_env.get_config().set("env.java.opts.client", "-XX:NativeMemoryTracking=summary")
(got the idea from the Tiny Flink presentation) but haven't had luck so far 1. Do all these things even work with Flink mini-cluster? 2. Am I setting things right? 3. Any way to check things are taking effect?
OK, so I don't think the setting
table_env.get_config().set("env.java.opts.all", "-XX:CompressedClassSpaceSize=100M -XX:MaxMetaspaceSize=200M")
was working. I can use
export JVM_ARGS="-XX:CompressedClassSpaceSize=100M -XX:MaxMetaspaceSize=200M"
, and it takes effect* (and fails the process sooner). I will try increasing the sizes. *When I say "it takes effect", it takes effect if I fix the way
JVM_ARGS
is passed; see https://github.com/apache/flink/pull/23500