I've installed plugin on JM and TM through common ...
# troubleshooting
j
I've installed plugin on JM and TM through common podtemplate init container, but still seeing this error to bring up flinksessionjob through jar from s3. DO we need to install plugin in Flink Kubernetes operator? If yes, how do we do that through values file for helm provided to install operator from community?
Copy code
Events:
  Type     Reason               Age                    From  Message
  ----     ------               ----                   ----  -------
  Warning  SessionJobException  3m32s (x117 over 77m)  Job   Could not find a file system implementation for scheme 's3'. The scheme is directly supported by Flink through the following plugin(s): flink-s3-fs-hadoop, flink-s3-fs-presto. Please ensure that each plugin resides within its own subfolder within the plugins directory. See <https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/plugins/> for more information. If you want to use a Hadoop file system for that scheme, please add the scheme to the configuration fs.allowed-fallback-filesystems. For a full list of supported file systems, please see <https://nightlies.apache.org/flink/flink-docs-stable/ops/filesystems/>.
m
j
Yes
Copy code
initContainers:
        - name: setup-certs-and-plugins
          image: localhost:5000/alpine:latest
          command:
            - "sh"
            - "-c"
            - "wget -O /opt/flink/plugins/flink-s3-fs-hadoop-1.20.2.jar <https://repo1.maven.org/maven2/org/apache/flink/flink-s3-fs-hadoop/1.20.2/flink-s3-fs-hadoop-1.20.2.jar> && wget -O /opt/flink/plugins/flink-s3-fs-presto-1.20.2.jar <https://repo1.maven.org/maven2/org/apache/flink/flink-s3-fs-presto/1.20.2/flink-s3-fs-presto-1.20.2.jar>"
          volumeMounts:
            - name: cert-volume
              mountPath: /usr/local/share/ca-certificates/
              readOnly: true
            - name: plugin-volume
              mountPath: "/opt/flink/plugins"
m
You are missing
s3-fs-presto
sub-directory
And I think
plugins
directory should not be under
/opt/flink
j
What should be the actual path in 1.20.x for plugins
We had this working for 1.16.x
Copy code
initContainers:
        - name: setup-certs-and-plugins
          image: "alpine:latest"
          command: ["sh", "-c", "wget -O /opt/flink/opt/flink-s3-fs-hadoop-1.16.1.jar <https://repo1.maven.org/maven2/org/apache/flink/flink-s3-fs-hadoop/1.16.1/flink-s3-fs-hadoop-1.16.1.jar> && wget -O /opt/flink/opt/flink-s3-fs-presto-1.16.1.jar <https://repo1.maven.org/maven2/org/apache/flink/flink-s3-fs-presto/1.16.1/flink-s3-fs-presto-1.16.1.jar>"]
          volumeMounts:
          - name: cert-volume
            mountPath: /usr/local/share/ca-certificates/
            readOnly: true
          - name: plugin-volume
            mountPath: "/opt/flink/opt"
      containers:
      - name: flink-main-container
        env: 
          - name: ENABLE_BUILT_IN_PLUGINS
            value: "flink-s3-fs-hadoop-1.16.1.jar;flink-s3-fs-presto-1.16.1.jar"
m
I don't think it changed in 1.20, you can either use env
ENABLE_BUILT_IN_PLUGINS
or copy plugins to
./plugins
directory. Those 2 plugins are already in the flink image, no need to download it. FYI: https://nightlies.apache.org/flink/flink-docs-release-1.20/docs/deployment/resource-providers/standalone/docker/#using-filesystem-plugins
j
I'm stuck with above error for plugin even though installed necessary plugin
it's blocking from running flinksessionjobs
Copy code
Job   Could not find a file system implementation for scheme 's3'. The scheme is directly supported by Flink through the following plugin(s): flink-s3-fs-hadoop, flink-s3-fs-presto. Please ensure that each plugin resides within its own subfolder within the plugins directory. See <https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/plugins/> for more information. If you want to use a Hadoop file system for that scheme, please add the scheme to the configuration fs.allowed-fallback-filesystems. For a full list of supported file systems, please see <https://nightlies.apache.org/flink/flink-docs-stable/ops/filesystems/>.
a
@Jashwanth S J are u able to solved the above issue ?
j
Key Point: For SessionJobs, the operator: 1. Fetches the jar to the operator pod's local filesystem first 2. Uploads it to the JobManager via REST API 3. JobManager then distributes it to TaskManagers
So, my experiment was correct and we should keep jars in operator image
a
Can u share how I add it in operator
i am using helm chart deployment
j
Take operator image from public repo and build a custom image using jenkins pipeline
Add a step to download your jars to the image in your Dockerfile
use the new image for operator deployment