Matt Camp
06/12/2024, 5:51 PMflink_config.yaml
│ File: /nix/store/cs50z9iwkp8zm2sa0y5h6yma14r209cg-flink-conf-drv/conf/flink-conf.yaml
───────┼──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
1 │ env.java.opts.all: --add-exports=java.base/sun.net.util=ALL-UNNAMED --add-exports=java.rmi/sun.rmi.registry=ALL-UNNAMED --add-exports=jdk.compiler/com.sun.tools.javac.api=ALL-UNNAMED --add-exports=jdk.
│ compiler/com.sun.tools.javac.file=ALL-UNNAMED --add-exports=jdk.compiler/com.sun.tools.javac.parser=ALL-UNNAMED --add-exports=jdk.compiler/com.sun.tools.javac.tree=ALL-UNNAMED --add-exports=jdk.compile
│ r/com.sun.tools.javac.util=ALL-UNNAMED --add-exports=java.security.jgss/sun.security.krb5=ALL-UNNAMED --add-opens=java.base/java.lang=ALL-UNNAMED --add-opens=java.base/java.net=ALL-UNNAMED --add-opens=
│ java.base/java.io=ALL-UNNAMED --add-opens=java.base/java.nio=ALL-UNNAMED --add-opens=java.base/sun.nio.ch=ALL-UNNAMED --add-opens=java.base/java.lang.reflect=ALL-UNNAMED --add-opens=java.base/java.text
│ =ALL-UNNAMED --add-opens=java.base/java.time=ALL-UNNAMED --add-opens=java.base/java.util=ALL-UNNAMED --add-opens=java.base/java.util.concurrent=ALL-UNNAMED --add-opens=java.base/java.util.concurrent.at
│ omic=ALL-UNNAMED --add-opens=java.base/java.util.concurrent.locks=ALL-UNNAMED
2 │ jobmanager.rpc.address: localhost
3 │ jobmanager.rpc.port: 6123
4 │ jobmanager.bind-host: 0.0.0.0
5 │ jobmanager.memory.process.size: 1600m
6 │ taskmanager.bind-host: 0.0.0.0
7 │ taskmanager.host: localhost
8 │ taskmanager.memory.process.size: 1728m
9 │ taskmanager.numberOfTaskSlots: 1
10 │ parallelism.default: 1
11 │ jobmanager.execution.failover-strategy: region
12 │ rest.address: localhost
13 │ rest.port: 8081
14 │ rest.bind-address: 0.0.0.0
15 │ env.log.dir: /tmp/flink-logs
16 │ env.java.home: /nix/store/sp9pjibkin0f5v48m648vfzizlg1pbfp-openjdk-11.0.19+7
17 │ env.path: /nix/store/8ll047q4iwk3sp73knx5ziph52la8f0h-python3-3.11.8-env/bin/:$PATH
18 │ python.path: /nix/store/8ll047q4iwk3sp73knx5ziph52la8f0h-python3-3.11.8-env/lib/python3.11/site-packages
19 │ python.executable: /nix/store/8ll047q4iwk3sp73knx5ziph52la8f0h-python3-3.11.8-env/bin/python
20 │ python.client.executable: /nix/store/8ll047q4iwk3sp73knx5ziph52la8f0h-python3-3.11.8-env/bin/python
21 │ metrics.reporter.prom.class: org.apache.flink.metrics.prometheus.PrometheusReporter
22 │ metrics.reporter.prom.port: 9249
23 │ pipeline.jars: local:///nix/store/w2lkz2q7l7zyvp8kr9vfnl9qm0x4wr69-flink-sql-connector-kafka-1.15.4.jar
do I still get this error?
╰─🚧 Traceback (most recent call last):
File "/nix/store/p50bpznylwb86lsrirzbdis6cm0bbyrk-example-flink-job/job/job.py", line 75, in <module>
run_example_flink_job(env, broker)
File "/nix/store/p50bpznylwb86lsrirzbdis6cm0bbyrk-example-flink-job/job/job.py", line 48, in run_example_flink_job
kafka_consumer = FlinkKafkaConsumer(
^^^^^^^^^^^^^^^^^^^
File "/nix/store/xyb7skbyksdjhynbgg4ks15yvqd8vr15-flink-1.18.0/opt/flink/opt/python/pyflink.zip/pyflink/datastream/connectors/kafka.py", line 203, in __init__
File "/nix/store/xyb7skbyksdjhynbgg4ks15yvqd8vr15-flink-1.18.0/opt/flink/opt/python/pyflink.zip/pyflink/datastream/connectors/kafka.py", line 161, in _get_kafka_consumer
File "/nix/store/xyb7skbyksdjhynbgg4ks15yvqd8vr15-flink-1.18.0/opt/flink/opt/python/pyflink.zip/pyflink/util/exceptions.py", line 185, in wrapped_call
TypeError: Could not found the Java class 'org.apache.flink.streaming.connectors.kafka.FlinkKafkaConsumer'. The Java dependencies could be specified via command line argument '--jarfile' or the config option 'pipeline.jars'
org.apache.flink.client.program.ProgramAbortException: java.lang.RuntimeException: Python process exits with code: 1
at org.apache.flink.client.python.PythonDriver.main(PythonDriver.java:140)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:355)
at org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:222)
at org.apache.flink.client.ClientUtils.executeProgram(ClientUtils.java:105)
at org.apache.flink.client.cli.CliFrontend.executeProgram(CliFrontend.java:851)
at org.apache.flink.client.cli.CliFrontend.run(CliFrontend.java:245)
at org.apache.flink.client.cli.CliFrontend.parseAndRun(CliFrontend.java:1095)
at org.apache.flink.client.cli.CliFrontend.lambda$mainInternal$9(CliFrontend.java:1189)
at org.apache.flink.runtime.security.contexts.NoOpSecurityContext.runSecured(NoOpSecurityContext.java:28)
at org.apache.flink.client.cli.CliFrontend.mainInternal(CliFrontend.java:1189)
at org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:1157)
Caused by: java.lang.RuntimeException: Python process exits with code: 1
at org.apache.flink.client.python.PythonDriver.main(PythonDriver.java:130)
... 14 moreMatt Camp
06/12/2024, 5:52 PMjar have to be an exact name?Matt Camp
06/12/2024, 5:52 PMMatt Camp
06/12/2024, 6:30 PMCLASSPATH to include the location of the kafka connector jar..Matt Camp
06/12/2024, 6:37 PMMatt Camp
06/12/2024, 6:38 PM╰─ stat /nix/store/w2lkz2q7l7zyvp8kr9vfnl9qm0x4wr69-flink-sql-connector-kafka-1.15.4.jar
File: /nix/store/w2lkz2q7l7zyvp8kr9vfnl9qm0x4wr69-flink-sql-connector-kafka-1.15.4.jar
Size: 5179763 Blocks: 9586 IO Block: 131072 regular file
Device: 0,29 Inode: 15312982 Links: 9
Access: (0444/-r--r--r--) Uid: ( 0/ root) Gid: ( 0/ root)
Access: 2024-06-12 13:33:03.359919447 -0500
Modify: 1969-12-31 18:00:01.000000000 -0600
Change: 2024-06-12 13:33:04.887927304 -0500
Birth: 2024-06-06 14:22:05.787510054 -0500