https://flink.apache.org/ logo
Join Slack
Powered by
Hello everyone, what is the correct way to use the...
# troubleshooting
e
Hello everyone, what is the correct way to use the kinesis-sql connector if I'm using flink with python on native K8s with EKS and deploying using this FlinkDeployment.yaml based from apache's flink kubernetes operator repo here? I've tried including the kinesis-sql-connect.jar in the image used and call it in the python script executed. But it keeps telling me 
Could not find any factory for identifier 'kinesis' that implements 'org.apache.flink.table.factories.DynamicTableFactory' in the classpath.
Copy code
apiVersion: <http://flink.apache.org/v1beta1|flink.apache.org/v1beta1>
kind: FlinkDeployment
metadata:
  name: python-kinesis-smoke
spec:
#I made this image, the dockerfile is in the thread of this comment
  image: <my_docker_hub_repo>/pyflink_kinesis:latest
  flinkVersion: v1_16
  flinkConfiguration:
    taskmanager.numberOfTaskSlots: "1"
  serviceAccount: flink
  jobManager:
    resource:
      memory: "2048m"
      cpu: 1
  taskManager:
    resource:
      memory: "2048m"
      cpu: 1
  job:
    jarURI: local:///opt/flink/opt/flink-python_2.12-1.16.1.jar # Note, this jarURI is actually a placeholder
    entryClass: "org.apache.flink.client.python.PythonDriver"
    args: ["-pyclientexec", "/usr/local/bin/python3", "-py", "/opt/flink/usrlib/python_demo.py"]
    parallelism: 1
    upgradeMode: stateless
In that image used ( 
pyflink_kinesis
) I deployed flink on a docker and put my python.py script as well as the 
flink-sql-connector-kinesis-1.16.2.jar
file. And in my python script I am calling this right after opening my table environment called t_env
Copy code
CURRENT_DIR = os.path.dirname(os.path.realpath(__file__))
    t_env.get_config().get_configuration().set_string(
        "pipeline.jars",
        "file:///" + CURRENT_DIR + "/lib/flink-sql-connector-kinesis-1.16.2.jar",
    )
This is the dockerfile I used to build that image:
Copy code
FROM flink:1.16

# install python3: it has updated Python to 3.9 in Debian 11 and so install Python 3.7 from source, \
# it currently only supports Python 3.6, 3.7 and 3.8 in PyFlink officially.

RUN apt-get update -y && \
apt-get install -y build-essential libssl-dev zlib1g-dev libbz2-dev libffi-dev && \
wget <https://www.python.org/ftp/python/3.7.9/Python-3.7.9.tgz> && \
tar -xvf Python-3.7.9.tgz && \
cd Python-3.7.9 && \
./configure --without-tests --enable-shared && \
make -j6 && \
make install && \
ldconfig /usr/local/lib && \
cd .. && rm -f Python-3.7.9.tgz && rm -rf Python-3.7.9 && \
ln -s /usr/local/bin/python3 /usr/local/bin/python && \
apt-get clean && \
rm -rf /var/lib/apt/lists/*

# install PyFlink
RUN pip3 install "apache-flink>=1.16.0,<1.17.1"


# add python script
USER flink
RUN mkdir /opt/flink/usrlib
ADD python_demo_kinesis.py /opt/flink/usrlib/python_demo.py

#Inside lib directory I have the flink-sql-connector-kinesis-1.16.2.jar
ADD lib /opt/flink/usrlib/lib
I am running this on EKS, and at least right now the error is about not adding correctly the kinesis dependency not anything about autentification for now
2 Views
Open in Slack
PreviousNext
Powered by