why when I have this `flink_config.yaml` ``` ...
# troubleshooting
m
why when I have this
flink_config.yaml
Copy code
│ File: /nix/store/cs50z9iwkp8zm2sa0y5h6yma14r209cg-flink-conf-drv/conf/flink-conf.yaml
───────┼──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
   1   │ env.java.opts.all: --add-exports=java.base/sun.net.util=ALL-UNNAMED --add-exports=java.rmi/sun.rmi.registry=ALL-UNNAMED --add-exports=jdk.compiler/com.sun.tools.javac.api=ALL-UNNAMED --add-exports=jdk.
       │ compiler/com.sun.tools.javac.file=ALL-UNNAMED --add-exports=jdk.compiler/com.sun.tools.javac.parser=ALL-UNNAMED --add-exports=jdk.compiler/com.sun.tools.javac.tree=ALL-UNNAMED --add-exports=jdk.compile
       │ r/com.sun.tools.javac.util=ALL-UNNAMED --add-exports=java.security.jgss/sun.security.krb5=ALL-UNNAMED --add-opens=java.base/java.lang=ALL-UNNAMED --add-opens=java.base/java.net=ALL-UNNAMED --add-opens=
       │ java.base/java.io=ALL-UNNAMED --add-opens=java.base/java.nio=ALL-UNNAMED --add-opens=java.base/sun.nio.ch=ALL-UNNAMED --add-opens=java.base/java.lang.reflect=ALL-UNNAMED --add-opens=java.base/java.text
       │ =ALL-UNNAMED --add-opens=java.base/java.time=ALL-UNNAMED --add-opens=java.base/java.util=ALL-UNNAMED --add-opens=java.base/java.util.concurrent=ALL-UNNAMED --add-opens=java.base/java.util.concurrent.at
       │ omic=ALL-UNNAMED --add-opens=java.base/java.util.concurrent.locks=ALL-UNNAMED
   2   │ jobmanager.rpc.address: localhost
   3   │ jobmanager.rpc.port: 6123
   4   │ jobmanager.bind-host: 0.0.0.0
   5   │ jobmanager.memory.process.size: 1600m
   6   │ taskmanager.bind-host: 0.0.0.0
   7   │ taskmanager.host: localhost
   8   │ taskmanager.memory.process.size: 1728m
   9   │ taskmanager.numberOfTaskSlots: 1
  10   │ parallelism.default: 1
  11   │ jobmanager.execution.failover-strategy: region
  12   │ rest.address: localhost
  13   │ rest.port: 8081
  14   │ rest.bind-address: 0.0.0.0
  15   │ env.log.dir: /tmp/flink-logs
  16   │ env.java.home: /nix/store/sp9pjibkin0f5v48m648vfzizlg1pbfp-openjdk-11.0.19+7
  17   │ env.path: /nix/store/8ll047q4iwk3sp73knx5ziph52la8f0h-python3-3.11.8-env/bin/:$PATH
  18   │ python.path: /nix/store/8ll047q4iwk3sp73knx5ziph52la8f0h-python3-3.11.8-env/lib/python3.11/site-packages
  19   │ python.executable: /nix/store/8ll047q4iwk3sp73knx5ziph52la8f0h-python3-3.11.8-env/bin/python
  20   │ python.client.executable: /nix/store/8ll047q4iwk3sp73knx5ziph52la8f0h-python3-3.11.8-env/bin/python
  21   │ metrics.reporter.prom.class: org.apache.flink.metrics.prometheus.PrometheusReporter
  22   │ metrics.reporter.prom.port: 9249
  23   │ pipeline.jars: local:///nix/store/w2lkz2q7l7zyvp8kr9vfnl9qm0x4wr69-flink-sql-connector-kafka-1.15.4.jar
do I still get this error?
Copy code
╰─🚧 Traceback (most recent call last):
  File "/nix/store/p50bpznylwb86lsrirzbdis6cm0bbyrk-example-flink-job/job/job.py", line 75, in <module>
    run_example_flink_job(env, broker)
  File "/nix/store/p50bpznylwb86lsrirzbdis6cm0bbyrk-example-flink-job/job/job.py", line 48, in run_example_flink_job
    kafka_consumer = FlinkKafkaConsumer(
                     ^^^^^^^^^^^^^^^^^^^
  File "/nix/store/xyb7skbyksdjhynbgg4ks15yvqd8vr15-flink-1.18.0/opt/flink/opt/python/pyflink.zip/pyflink/datastream/connectors/kafka.py", line 203, in __init__
  File "/nix/store/xyb7skbyksdjhynbgg4ks15yvqd8vr15-flink-1.18.0/opt/flink/opt/python/pyflink.zip/pyflink/datastream/connectors/kafka.py", line 161, in _get_kafka_consumer
  File "/nix/store/xyb7skbyksdjhynbgg4ks15yvqd8vr15-flink-1.18.0/opt/flink/opt/python/pyflink.zip/pyflink/util/exceptions.py", line 185, in wrapped_call
TypeError: Could not found the Java class 'org.apache.flink.streaming.connectors.kafka.FlinkKafkaConsumer'. The Java dependencies could be specified via command line argument '--jarfile' or the config option 'pipeline.jars'
org.apache.flink.client.program.ProgramAbortException: java.lang.RuntimeException: Python process exits with code: 1
	at org.apache.flink.client.python.PythonDriver.main(PythonDriver.java:140)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:355)
	at org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:222)
	at org.apache.flink.client.ClientUtils.executeProgram(ClientUtils.java:105)
	at org.apache.flink.client.cli.CliFrontend.executeProgram(CliFrontend.java:851)
	at org.apache.flink.client.cli.CliFrontend.run(CliFrontend.java:245)
	at org.apache.flink.client.cli.CliFrontend.parseAndRun(CliFrontend.java:1095)
	at org.apache.flink.client.cli.CliFrontend.lambda$mainInternal$9(CliFrontend.java:1189)
	at org.apache.flink.runtime.security.contexts.NoOpSecurityContext.runSecured(NoOpSecurityContext.java:28)
	at org.apache.flink.client.cli.CliFrontend.mainInternal(CliFrontend.java:1189)
	at org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:1157)
Caused by: java.lang.RuntimeException: Python process exits with code: 1
	at org.apache.flink.client.python.PythonDriver.main(PythonDriver.java:130)
	... 14 more
does the
jar
have to be an exact name?
it works if i pass the command line arg but I dont want to do that I want to have it in the config
I've also tried setting my
CLASSPATH
to include the location of the kafka connector jar..
the job and task manager see the file or are atleast aware of the config with the option set
Copy code
╰─ stat /nix/store/w2lkz2q7l7zyvp8kr9vfnl9qm0x4wr69-flink-sql-connector-kafka-1.15.4.jar                              
  File: /nix/store/w2lkz2q7l7zyvp8kr9vfnl9qm0x4wr69-flink-sql-connector-kafka-1.15.4.jar
  Size: 5179763   	Blocks: 9586       IO Block: 131072 regular file
Device: 0,29	Inode: 15312982    Links: 9
Access: (0444/-r--r--r--)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2024-06-12 13:33:03.359919447 -0500
Modify: 1969-12-31 18:00:01.000000000 -0600
Change: 2024-06-12 13:33:04.887927304 -0500
 Birth: 2024-06-06 14:22:05.787510054 -0500