https://flink.apache.org/ logo
Join Slack
Powered by
Hello everyone! I have a question, does anyone kn...
# troubleshooting
e
Hello everyone! I have a question, does anyone knows if the k8s operator supports pyflink jobs? I have found some demo examples and they work, but now I have tried researching thoroughly how to include connectors (like kafka/kinesis/jdbc/mongoDB/etc..) for pyflink on k8s operator across flink's docs or other blogs/repos. And I have seen that other folks are experiencing that documentation for deploying pyflink is hard to find (like this article) and it uses CLI submission rather than the operator. And this unanswered stackoverflow question about how to deploy a scalable pyflink, which I would love to answer saying there's a k8s operator, if only I knew how to include connectors to the k8s operator 🙂 . So maybe I thought here I could ask and learn? 😄 My case is that I have already deployed a pyflink with k8s operator 
FlinkDeployment
, and it works well if I sink to console, based from this repo example. But I am currently just stuck on trying to add a source/sink .jar connector, which I think it might be a very simple java thing I can't see, like some .pom file. I've used those dependencies on local pyflink before with no problems, but on k8s operator they don't seem to be found by python even though it does locally with no problems. Maybe it's a simple java thing I am not seeing? (I don't know java) I would also love to help to document how I did it for other future folks that might come to the same question, if only I knew how to include the connectors .jar in pyflink k8s operator. I think the least I could do in that regard is publishing this stackoverflow question with more details on how I am doing it, but it's still unfinished as it's asking about the connectors. I think it's very close to work and probably I am missing just some small java detail I can't find to make it work. I would love if someone could help me including connector's dependencies to pyflink with the k8s operator. I would be very grateful!
h
e
Hi @Huib thanks a lot for the support! Yes that's the one I am basing on and it works perfectly, but it sinks to console so I am just trying to modify it to sink to kinesis of kafka or something, but just adding the connector part is the one I can't get working. For example I have used the kinesis connector.jar and pointed python to use it, and used a kinesis sink locally and it works. But when I deploy it with the k8s operator (even though I also upload my new docker image with the connecto .jar on it) I get the error that Kinesis is not recognized in the Factory
For example, this is the python code snippet I made inside my image that points to the connector (also included in the image on that /lib path the code is referencing)
Copy code
t_env.get_config().get_configuration().set_string(
        "pipeline.jars",
        "file:///" + CURRENT_DIR + "/lib/flink-sql-connector-kinesis-1.16.2.jar",
    )
h
For me, putting the jars in 
/opt/flink/usrlib/
made flink pick them up, where did you put yours?
but that could be due to how my dockerfile is set up, can you share yours?
e
Oh okay I've set them here 
/opt/flink/usrlib/lib
Sure thing!, this is my dockerfile
Copy code
FROM flink:1.16

# install python3: it has updated Python to 3.9 in Debian 11 and so install Python 3.7 from source, \
# it currently only supports Python 3.6, 3.7 and 3.8 in PyFlink officially.

RUN apt-get update -y && \
apt-get install -y build-essential libssl-dev zlib1g-dev libbz2-dev libffi-dev && \
wget <https://www.python.org/ftp/python/3.7.9/Python-3.7.9.tgz> && \
tar -xvf Python-3.7.9.tgz && \
cd Python-3.7.9 && \
./configure --without-tests --enable-shared && \
make -j6 && \
make install && \
ldconfig /usr/local/lib && \
cd .. && rm -f Python-3.7.9.tgz && rm -rf Python-3.7.9 && \
ln -s /usr/local/bin/python3 /usr/local/bin/python && \
apt-get clean && \
rm -rf /var/lib/apt/lists/*

# install PyFlink
RUN pip3 install "apache-flink>=1.16.0,<1.17.1"


# add python script
USER flink
RUN mkdir /opt/flink/usrlib
ADD python_demo_kinesis.py /opt/flink/usrlib/python_demo.py

#Inside lib directory I have the flink-sql-connector-kinesis-1.16.2.jar
ADD lib /opt/flink/usrlib/lib
I based it from this one from the repo. I just added my .jar in my /lib directory into 
/opt/flink/usrlib/lib
. I also ran that image locally with docker and got in the bash to confirm the .jar was correctly located at 
/opt/flink/usrlib/lib
and it does
h
Can you try putting the jar in the /opt/flink/userlib folder, without the lib? I am 99% that’s what’s working for me 😉
so right next to your .py file
e
Ok sure 😄 !
omg I think it's running! @Huib
I'll check the stream
Okay it's not logging but it might be other thing now since I can see the flink dashboard now and the pod logs says it running with no more errors about not finding kinesis factory!
Thanks a lot!
h
Nice! Glad I could help 🙂
e
yes thanks a lot! I'll make some post about how to run it when I fix the rest now 😄
5 Views
Open in Slack
PreviousNext
Powered by