Deepyaman Datta
10/09/2023, 1:28 PM> raise Py4JJavaError(
"An error occurred while calling {0}{1}{2}.\n".
format(target_id, ".", name), value)
E py4j.protocol.Py4JJavaError: An error occurred while calling o8.sqlQuery.
E : java.lang.OutOfMemoryError: Compressed class space
I'm not sure what the fix is; something I found is https://lists.apache.org/thread/6ls2m9oslj5w9l9rojfv22nskp635hmy. Maybe https://stackoverflow.com/questions/62879157/usecompressedoops-usecompressedclasspointers-in-jdk-13-and-jdk-15 could be something else to look into.
However, I'm not sure what I'm doing is effective. I've tried, when creating the table environment with PyFlink, all sorts of combinations of:
table_env.get_config().set("env.java.opts.all", "-XX:CompressedClassSpaceSize=100M -XX:MaxMetaspaceSize=200M") # <https://lists.apache.org/thread/6ls2m9oslj5w9l9rojfv22nskp635hmy>
as well as
table_env.get_config().set("parallelism.default", "1")
but things still fail.
I also tried to debug with:
table_env.get_config().set("env.java.opts.all", "-XX:+PrintFlagsFinal")
table_env.get_config().set("env.java.opts.all", "-XX:NativeMemoryTracking=summary")
table_env.get_config().set("env.java.opts.client", "-XX:+PrintFlagsFinal")
table_env.get_config().set("env.java.opts.client", "-XX:NativeMemoryTracking=summary")
(got the idea from the Tiny Flink presentation)
but haven't had luck so far
1. Do all these things even work with Flink mini-cluster?
2. Am I setting things right?
3. Any way to check things are taking effect?Deepyaman Datta
10/09/2023, 5:43 PMtable_env.get_config().set("env.java.opts.all", "-XX:CompressedClassSpaceSize=100M -XX:MaxMetaspaceSize=200M") was working. I can use export JVM_ARGS="-XX:CompressedClassSpaceSize=100M -XX:MaxMetaspaceSize=200M", and it takes effect* (and fails the process sooner). I will try increasing the sizes.
*When I say "it takes effect", it takes effect if I fix the way JVM_ARGS is passed; see https://github.com/apache/flink/pull/23500