VISHAL B
10/09/2024, 1:38 PMroot@data-team-poc-instance-mum:~/flink# ./bin/flink run --python /root/pyflink/test.py
Traceback (most recent call last):
File "/root/pyflink/test.py", line 51, in module
main()
File "/root/pyflink/test.py", line 48, in main
env.execute("Flink Kafka Consumer")
File "/root/flink/opt/python/pyflink.zip/pyflink/datastream/stream_execution_environment.py", line 813, in execute
File "/root/flink/opt/python/py4j-0.10.9.7-src.zip/py4j/java_gateway.py", line 1322, in __call__
File "/root/flink/opt/python/pyflink.zip/pyflink/util/exceptions.py", line 146, in deco
File "/root/flink/opt/python/py4j-0.10.9.7-src.zip/py4j/protocol.py", line 326, in get_return_value
py4j.protocol.Py4JJavaError: An error occurred while calling o0.execute.
: java.net.MalformedURLException: no protocol: ['file:/root/flink/opt/flink-python-1.19.1.jar']
at java.base/java.net.URL.init(URL.java:645)
at java.base/java.net.URL.init(URL.java:541)
at java.base/java.net.URL.init(URL.java:488)
at org.apache.flink.configuration.ConfigUtils.decodeListFromConfig(ConfigUtils.java:133)
at org.apache.flink.client.cli.ExecutionConfigAccessor.getJars(ExecutionConfigAccessor.java:77)
at org.apache.flink.client.deployment.executors.PipelineExecutorUtils.getJobGraph(PipelineExecutorUtils.java:77)
at org.apache.flink.client.deployment.executors.AbstractSessionClusterExecutor.execute(AbstractSessionClusterExecutor.java:72)
at org.apache.flink.streaming.api.environment.StreamExecutionEnvironment.executeAsync(StreamExecutionEnvironment.java:2440)
at org.apache.flink.client.program.StreamContextEnvironment.executeAsync(StreamContextEnvironment.java:188)
at org.apache.flink.client.program.StreamContextEnvironment.execute(StreamContextEnvironment.java:117)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at org.apache.flink.api.python.shaded.py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
at org.apache.flink.api.python.shaded.py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:374)
at org.apache.flink.api.python.shaded.py4j.Gateway.invoke(Gateway.java:282)
at org.apache.flink.api.python.shaded.py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
at org.apache.flink.api.python.shaded.py4j.commands.CallCommand.execute(CallCommand.java:79)
at org.apache.flink.api.python.shaded.py4j.GatewayConnection.run(GatewayConnection.java:238)
at java.base/java.lang.Thread.run(Thread.java:829)
org.apache.flink.client.program.ProgramAbortException: java.lang.RuntimeException: Python process exits with code: 1
at org.apache.flink.client.python.PythonDriver.main(PythonDriver.java:134)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:355)
at org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:222)
at org.apache.flink.client.ClientUtils.executeProgram(ClientUtils.java:108)
at org.apache.flink.client.cli.CliFrontend.executeProgram(CliFrontend.java:1026)
at org.apache.flink.client.cli.CliFrontend.run(CliFrontend.java:247)
at org.apache.flink.client.cli.CliFrontend.parseAndRun(CliFrontend.java:1270)
at org.apache.flink.client.cli.CliFrontend.lambda$mainInternal$10(CliFrontend.java:1367)
at org.apache.flink.runtime.security.contexts.NoOpSecurityContext.runSecured(NoOpSecurityContext.java:28)
at org.apache.flink.client.cli.CliFrontend.mainInternal(CliFrontend.java:1367)
at org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:1335)
Caused by: java.lang.RuntimeException: Python process exits with code: 1
at org.apache.flink.client.python.PythonDriver.main(PythonDriver.java:124)Aleksandr Pilipenko
10/09/2024, 1:51 PMjava.net.MalformedURLException: no protocol: ['file:/root/flink/opt/flink-python-1.19.1.jar']
do you have an array with jars that is not unwrapped? path should be file:/... , not ['file:/...']VISHAL B
10/09/2024, 1:58 PMfrom pyflink.common import SimpleStringSchema,WatermarkStrategy
from pyflink.datastream import StreamExecutionEnvironment
from pyflink.datastream.connectors import KafkaSource
from pyflink.datastream.connectors.kafka import KafkaOffsetsInitializer
def main():
# Create the execution environment
env = StreamExecutionEnvironment.get_execution_environment()
# Add the Kafka connector JAR
# env.get_configuration().set_string("pipeline.jars", "file:///root/pyflink/connectorkafka.jar")
env.add_jars("file:///root/pyflink/connectorkafka.jar")
env.add_jars("file:///root/pyflink/flink-connector-starrocks-1.2.9_flink-1.18.jar")
brokers = "localhost:9092"
source = KafkaSource.builder() \
.set_bootstrap_servers(brokers) \
.set_topics("dbserver1.public.collection_requests") \
.set_group_id("test-group") \
.set_starting_offsets(KafkaOffsetsInitializer.earliest()) \
.set_value_only_deserializer(SimpleStringSchema()) \
.build()
stream = env.from_source(source, WatermarkStrategy.no_watermarks(), "Kafka Source")
stream.print() # This will print the messages to the console
# Execute the Flink job
env.execute("Flink Kafka Consumer")
if __name__ == "__main__":
main()VISHAL B
10/09/2024, 1:59 PMAleksandr Pilipenko
10/09/2024, 11:47 PMadd_jars method.
As a potential workaround, you should be able to set this configuration with .set_string("pipeline.jars", "<list of jars>") like you do in the table API example, however you’ll probably need to also add flink-python there as well (path that you see in exception from your original question).