https://flink.apache.org/ logo
Join Slack
Powered by
Ensure the flink operator and image running the de...
# troubleshooting
h
Ensure the flink operator and image running the deployment job both have the plug-in enabled . This is for the operator FROM apache/flink-kubernetes-operator # Download the S3 Presto plugin RUN mkdir -p /opt/flink/plugins/s3-fs-presto RUN wget https://repo1.maven.org/maven2/org/apache/flink/flink-s3-fs-presto/1.17.1/flink-s3-fs-presto-1.17.1.jar -O /opt/flink/plugins/s3-fs-presto/flink-s3-fs-presto-1.17.1.jar # Set the environment variable to enable the plugin ENV ENABLE_BUILT_IN_PLUGINS=flink-s3-fs-presto-1.17.1.jar
a
ok .. will try this out
@Howard Hill I still keep getting the same error :/ As you can see now, I have set the env variable as well now
Copy code
apiVersion: <http://flink.apache.org/v1beta1|flink.apache.org/v1beta1>
kind: FlinkDeployment
metadata:
  name: basic-example
spec:
  image: flink:1.17
  flinkVersion: v1_17
  flinkConfiguration:
    taskmanager.numberOfTaskSlots: "2"
  serviceAccount: flink
  jobManager:
    resource:
      memory: "1024m"
      cpu: 1
  taskManager:
    resource:
      memory: "1024m"
      cpu: 1
  job:
    jarURI: local:///opt/flink/examples/streaming/StateMachineExample.jar
    parallelism: 1
    upgradeMode: stateless
This is the flinkdeployment manifest that I am using. As per your initial response, are you saying that I need to modify this base image as well- flink:1.17 ? Do I need to copy the jar file in this image as well or just enable the ENV NABLE_BUILT_IN_PLUGINS
As of now, ive just been modifying the operator image
a
you have to add the ENABLE_BUILT_IN_PLUGINS in podTemplate.spec.containers
Copy code
apiVersion: <http://flink.apache.org/v1beta1|flink.apache.org/v1beta1>
kind: FlinkDeployment
metadata:
  name: basic-example
spec:
  image: flink:1.17
  flinkVersion: v1_17
  flinkConfiguration:
    taskmanager.numberOfTaskSlots: "2"
  serviceAccount: flink
  podTemplate:
    apiVersion: v1
    kind: Pod
    metadata:
      name: pod-template
    spec:
      containers:
        - name: flink-main-container
          env:
             - name: ENABLE_BUILT_IN_PLUGINS
              value: flink-s3-fs-presto-1.17.1.jar
  jobManager:
    resource:
      memory: "1024m"
      cpu: 1
  taskManager:
    resource:
      memory: "1024m"
      cpu: 1
  job:
    jarURI: local:///opt/flink/examples/streaming/StateMachineExample.jar
    parallelism: 1
    upgradeMode: stateless
As far as I know, if your Flink pipeline just stores checkpoint in S3, it might not need to use 
s3p://
, 
s3://
should be sufficient
2 Views
Open in Slack
PreviousNext
Powered by