hello everyone, I am trying to create Flink Deploy...
# troubleshooting
m
hello everyone, I am trying to create Flink Deployment and FlinkSessionJob. My application is working well when i create deployment only. But i have a problem with jar files Here is my yaml file;
Copy code
apiVersion: <http://flink.apache.org/v1beta1|flink.apache.org/v1beta1>
kind: FlinkDeployment
metadata:
  name: test
  namespace: flink
spec:
  ingress:
    template:  "{{name}}.<http://test.com|test.com>"
    className: "nginx"
    annotations: {}
  image: <private pyflink repo>
  imagePullPolicy: "Always"
  flinkVersion: v1_16
  flinkConfiguration:
    taskmanager.numberOfTaskSlots: "1"
  serviceAccount: flink
  jobManager:
    podTemplate:
      apiVersion: v1
      kind: Pod
      metadata:
       name: pod-template
      spec:
        serviceAccount: flink
        containers:
        # Do not change the main container name
          - name: flink-main-container
            envFrom:
            - secretRef:
                name: flink-secret
    resource:
      memory: "2048m"
      cpu: 1
  taskManager:
    resource:
      memory: "2048m"
      cpu: 1
    replicas: 1

---

apiVersion: <http://flink.apache.org/v1beta1|flink.apache.org/v1beta1>
kind: FlinkSessionJob
metadata:
  name: rabbitmq-to-kafka
  namespace: flink
spec:
  deploymentName: test
  job:
    jarURI: local:///opt/flink/lib/flink-sql-connector-rabbitmq-1.16.0.jar;local:///opt/flink/lib/flink-sql-connector-kafka-1.16.0.jar
    entryClass: "org.apache.flink.client.python.PythonDriver"
    args: ["-pyclientexec", "/usr/local/bin/python3", "-py", "/opt/flink/usrlib/rabbitmq_to_kafka.py"]
    parallelism: 4
    upgradeMode: stateless
    
---
apiVersion: <http://flink.apache.org/v1beta1|flink.apache.org/v1beta1>
kind: FlinkSessionJob
metadata:
  name: kafka-to-s3-raw
  namespace: flink
spec:
  deploymentName: test
  job:
    jarURI: local:///opt/flink/lib/flink-sql-connector-rabbitmq-1.16.0.jar;local:///opt/flink/lib/flink-sql-connector-kafka-1.16.0.jar
    entryClass: "org.apache.flink.client.python.PythonDriver"
    args: ["-pyclientexec", "/usr/local/bin/python3", "-py", "/opt/flink/usrlib/kafka_to_s3_raw.py"]
    parallelism: 4
    upgradeMode: stateless
I am getting ” resource id: ResourceID{name=‘kafka-to-s3-raw’, namespace=‘flink’}, version: 6850788} failed. org.apache.flink.kubernetes.operator.exception.ReconciliationException: org.apache.flink.core.fs.UnsupportedFileSystemSchemeException: Could not find a file system implementation for scheme ‘local’. The scheme is not directly supported by Flink and no Hadoop file system to support this scheme could be loaded. For a full list of supported file systems, please see https://nightlies.apache.org/flink/flink-docs-stable/ops/filesystems/.” It is working well when i put jobURI inside of deployment. I tried different alternatives but didn’t figure it out. Can you help me about this error ? Many thanks.
k
flinksessionjob's jarURI filesystem you provided wont work, please the other types like http
See below example for submitting the jobs
Copy code
apiVersion: <http://flink.apache.org/v1beta1|flink.apache.org/v1beta1>
kind: FlinkSessionJob
metadata:
  name: basic-session-job-only-example
spec:
  deploymentName: basic-session-deployment-only-example
  job:
    jarURI: <https://repo1.maven.org/maven2/org/apache/flink/flink-examples-streaming_2.12/1.15.3/flink-examples-streaming_2.12-1.15.3-TopSpeedWindowing.jar>
    parallelism: 4
    upgradeMode: stateless
m
Hi @kingsathurthi How can i append multiple jars url in jarURI
I tried to write with “;” separated but it didn’t work. For example: https://xxxx.jar;https://yyy.jar
k
use two flinksessionjob to submitting your jar like below
Copy code
---

apiVersion: <http://flink.apache.org/v1beta1|flink.apache.org/v1beta1>
kind: FlinkSessionJob
metadata:
  name: rabbitmq-to-kafka
  namespace: flink
spec:
  deploymentName: test
  job:
    jarURI: <https://registry.com/job_1.jar>
    entryClass: "org.apache.flink.client.python.PythonDriver"
    args: ["-pyclientexec", "/usr/local/bin/python3", "-py", "/opt/flink/usrlib/rabbitmq_to_kafka.py"]
    parallelism: 4
    upgradeMode: stateless
    
---
apiVersion: <http://flink.apache.org/v1beta1|flink.apache.org/v1beta1>
kind: FlinkSessionJob
metadata:
  name: kafka-to-s3-raw
  namespace: flink
spec:
  deploymentName: test
  job:
    jarURI: <https://registry.com/job_2.jar>
    entryClass: "org.apache.flink.client.python.PythonDriver"
    args: ["-pyclientexec", "/usr/local/bin/python3", "-py", "/opt/flink/usrlib/kafka_to_s3_raw.py"]
    parallelism: 4
    upgradeMode: stateless
m
Actually i dont wanna submit job. I want to add kafka and rabbit mq jars for each Session job