https://flink.apache.org/ logo
Join Slack
Powered by
Hi all! I'm constantly getting an 's3:// scheme no...
# troubleshooting
e
Hi all! I'm constantly getting an 's3:// scheme not supported' error when trying to deploy a FlinkSessionJob. I'm currently using a session deployment. One JobManager + TaskManager with a FlinkDeployment, all using the kubernetes-operators. P.S.: Adding the s3-fs-presto plugin to the plugins /opt/flink/plugins folder (in the podTemplate FlinkDeployment.yml) did not help. Has anyone faced that before? Any guidance will be appreciated. I've been stuck on it for 24hours now
r
e
Not really! But I'll see if I can use the same debugging steps. Which begs the question, between the job manager and the Kubernetes operator, who handles the jar download?
r
i've not used k8s at all. but iiuc you have to provide the JAR in the right location.
(they don't download it)
e
When using an HTTPS link, it works fine; I guess it's downloaded from the URL
apiVersion: <http://flink.apache.org/v1beta1|flink.apache.org/v1beta1>
kind: FlinkSessionJob
metadata:
name: flink-job
spec:
job:
jarURI: file:///opt/flink/jars/flink-examples-streaming_2.12-1.16.1-TopSpeedWindowing.jar
parallelism: 1
state: running
deploymentName: flink-jobmanager
The jarURI, that's where I'm bumping
I need to use s3:// there
It's in the doc
But doesnt work as explained
a
seems like he also running into the same issues, seems like we have to put jar files in the operator image
I also need to deploy it, if I get it working will share the docker and manifest files
e
Ok, great. Let's hope it works.
a
I also faced the same issue able to resolve it now let me know if u still need some help https://apache-flink.slack.com/archives/C03G7LJTS2G/p1764074908329699?thread_ts=1763447091.430249&amp;cid=C03G7LJTS2G
No need to put jar in operator . I am able to deploy session job with s3 url.
a
@Anuj Jain are you submitting jobs using pyflink or jar files, am new to flink, will the base deployment need change if I want to use pyflink
@rmoff thank you soo much for the link, saw other blogs in your profile they are going to save me soo much time for me
flink 1
a
I am using jar based deployment but I do not think that is related to the s3 plugin issue.
in my operator values.yaml i defined an init container to load plugin , same you have to define in pod template for job-manager/task-manager.
Copy code
# optional init containers for operator pod
  initContainers:
    - name: download-s3-plugins
      image: c.rzp.io/proxy_dockerhub/busybox:1.35.0
      command:
        - /bin/sh
        - -c
        - |
          set -e
          echo "🔧 Downloading S3 plugins for Flink Operator (2.1.1)..."
          mkdir -p /opt/flink/plugins/s3-fs-hadoop
          mkdir -p /opt/flink/plugins/s3-fs-presto

          echo "Downloading flink-s3-fs-hadoop-2.1.1.jar..."
          wget -O /opt/flink/plugins/s3-fs-hadoop/flink-s3-fs-hadoop-1.20.0.jar \
            <https://repo1.maven.org/maven2/org/apache/flink/flink-s3-fs-hadoop/1.20.0/flink-s3-fs-hadoop-1.20.0.jar>

          echo "Downloading flink-s3-fs-presto-2.1.1.jar..."
          wget -O /opt/flink/plugins/s3-fs-presto/flink-s3-fs-presto-1.20.0.jar \
            <https://repo1.maven.org/maven2/org/apache/flink/flink-s3-fs-presto/1.20.0/flink-s3-fs-presto-1.20.0.jar>

          echo "S3 plugins downloaded successfully!"
          echo "Plugin files:"
          ls -lh /opt/flink/plugins/s3-fs-hadoop/
          ls -lh /opt/flink/plugins/s3-fs-presto/
      volumeMounts:
        - name: flink-plugins
          mountPath: /opt/flink/plugins
You also need to define Access key and secret key env variable as iam role based access does not work with it. i am not sure why iam role based access not working.
l
r
I had to change the scheme to 'sa3://' but I was using the aws s3 plugin
a
@Ricardo Mendoza 
s3a://
?
l
Open in Slack
PreviousNext
Powered by