Bruno Filippone
06/20/2023, 4:34 PMjob.jarURI part of a session job, where I get the error: Could not find a file system implementation for scheme 's3'. What am I missing?
Here are my Kubernetes resources for context (shortened versions):
apiVersion: flink.apache.org/v1beta1
kind: FlinkDeployment
metadata:
name: session-cluster
namespace: flink-jobs
spec:
image: flink:1.17
flinkVersion: v1_17
jobManager:
[...]
podTemplate:
spec:
containers:
- name: flink-main-container
env:
- name: ENABLE_BUILT_IN_PLUGINS
value: flink-s3-fs-presto-1.17.1.jar;flink-s3-fs-hadoop-1.17.1.jar
taskManager:
[...]
podTemplate:
spec:
containers:
- name: flink-main-container
env:
- name: ENABLE_BUILT_IN_PLUGINS
value: flink-s3-fs-presto-1.17.1.jar;flink-s3-fs-hadoop-1.17.1.jar
serviceAccount: flink
---
apiVersion: flink.apache.org/v1beta1
kind: FlinkSessionJob
metadata:
name: session-cluster-job
namespace: flink-jobs
spec:
deploymentName: session-cluster
job:
jarURI: s3://flink-test-bucket/TopSpeedWindowing.jar
parallelism: 4
upgradeMode: statelessZhong Chen
06/20/2023, 4:42 PMRUN mkdir -p /opt/flink/plugins/s3-fs-hadoop
RUN \
wget -O /opt/flink/plugins/s3-fs-hadoop/flink-s3-fs-hadoop-$FLINK_VERSION.jar \
https://repo1.maven.org/maven2/org/apache/flink/flink-s3-fs-hadoop/$FLINK_VERSION/flink-s3-fs-hadoop-$FLINK_VERSION.jarBruno Filippone
06/20/2023, 4:43 PMBruno Filippone
06/20/2023, 4:45 PMZhong Chen
06/20/2023, 4:46 PMapplication mode for my jobs, and the jar files are placed during the image build process.Zhong Chen
06/20/2023, 4:47 PMFROM eclipse-temurin:11-jre-jammy AS build
# Disable Prompt During Packages Installation
ARG DEBIAN_FRONTEND=noninteractive
# Update Ubuntu Software repository
RUN apt update
RUN mkdir /flink-jobs
WORKDIR /flink-jobs
# Install Scala
RUN curl -fL https://github.com/coursier/coursier/releases/latest/download/cs-x86_64-pc-linux.gz | \
gzip -d > cs && chmod +x cs && ./cs setup --yes
RUN ./cs install scala:2.12.17 && ./cs install scalac:2.12.17
# Install sbt
ARG SBT_VERSION=1.8.2
RUN \
mkdir /working/ && \
cd /working/ && \
curl -L -o sbt-$SBT_VERSION.deb https://repo.scala-sbt.org/scalasbt/debian/sbt-$SBT_VERSION.deb && \
dpkg -i sbt-$SBT_VERSION.deb && \
rm sbt-$SBT_VERSION.deb && \
apt-get update && \
apt-get install sbt && \
cd && \
rm -r /working/ && \
sbt sbtVersion
# Clean up cache to reduce image size
RUN rm -rf /var/lib/apt/lists/*
RUN apt cleanBruno Filippone
06/20/2023, 4:47 PMZhong Chen
06/20/2023, 4:47 PMlocal directory in flink imageBruno Filippone
06/20/2023, 4:48 PMBruno Filippone
06/20/2023, 4:49 PMBruno Filippone
06/20/2023, 5:07 PMBruno Filippone
06/20/2023, 5:08 PMTo support jar from different filesystems, you should extend the base docker image as below, and put the related filesystem jar to the plugin dir and deploy the operator.
So I need a custom flink-kubernetes-operator Docker image with the s3 plugin.