Hi All, I'm Using Flink flink operator and deployed deployment only. then i tried submitting below session job with local filesystem as jarURI and im getting this error

apiVersion: <http://flink.apache.org/v1beta1|flink.apache.org/v1beta1>
kind: FlinkSessionJob
metadata:
  name: basic-session-job-only-example-2
spec:
  deploymentName: basic-session-deployment-only-example
  job:
    jarURI: local:///opt/flink/examples/streaming/StateMachineExample.jar
    parallelism: 4
    upgradeMode: stateless

Error :

Spec:

Deployment Name:  basic-session-deployment-only-example

Job:

Args:

Jar URI:       local:///opt/flink/examples/streaming/StateMachineExample.jar

Parallelism:   4

State:         running

Upgrade Mode:  stateless

Status:

Error:  {"type":"org.apache.flink.kubernetes.operator.exception.ReconciliationException","message":"org.apache.flink.core.fs.UnsupportedFileSystemSchemeException: Could not find a file s

ystem implementation for scheme 'local'. The scheme is not directly supported by Flink and no Hadoop file system to support this scheme could be loaded. For a full list of supported file s

ystems, please see <https://nightlies.apache.org/flink/flink-docs-stable/ops/filesystems/>.","throwableList":[{"type":"org.apache.flink.core.fs.UnsupportedFileSystemSchemeException","message

":"Could not find a file system implementation for scheme 'local'. The scheme is not directly supported by Flink and no Hadoop file system to support this scheme could be loaded. For a ful

l list of supported file systems, please see <https://nightlies.apache.org/flink/flink-docs-stable/ops/filesystems/>."},{"type":"org.apache.flink.core.fs.UnsupportedFileSystemSchemeException

","message":"Hadoop is not in the classpath/dependencies."}]}

If the jar is present on the operator side you can simply use

file://

So I cannot use the inbuilt example jar available in Job manager pod?

wait, you cant do that

Operator you mean? Is Flink operator pod?

yes

Oh, why like that? It would be great if session job take the jar from deployment

This is how Flink Session jobs work unfortunately. The operator cannot change this behaviour 😕

Can you point some example?

I don’t have any sorry

Okay let me try add class path in deployment

Hey @kingsathurthi did this work for you ? I am also kinda in a similar situation.

No, flinksession supports file:// flinkdeployment support local://

okay thanks. I was also trying to load the jar remotely from s3 in the flinksessionjob. I have got the s3-fs-hadoop dependency in plugins folder of the operator and also the credential provider is set. However, when I am submitting the job, it is stuck in UPGRADING stage and it doesn’t throw any error or exception anywhere. We are still trying to understand the behaviour. Please Lmk if you have tried this or encountered a similar scenario ! Thanks

Kubectl describe flinkdep <name> Kubectl describe flinksessionjob <name> Above command will show errors related flink jobs/deployment

Yes did that but there are no error logs in flink operator nor events in the cluster to understand more. Howeveer, upgrading the flink operator version to 1.5.0 worked locally in the minikube cluster FYI. I am trying to upgrade it now in the env cluster.

Yep may be, 1.5.0 has lot of improvement

yes doing that too ! thanks 🙂

That's a good tip in general we are always working on improvements bug fixes and best to use the latest :)

FYI @Gyula Fóra @kingsathurthi upgrading the operator to 1.5.0 worked. The operator is now able to download the jar the bucket. However, one small catch here is we have configured the s3.accessKey and s3.secretKey in the flink-conf.yaml of the operator. Do you know if we can configure the operator to make use of the service account ? (we have a service account which as IAM role which has access to S3 operations.)

Gyula may help on this,

@Gyula Fóra any clue !?

I dont really know sorry 😞