Hi everyone, I am new to flink and using these libraries.

apache-flink==1.14.4
apache-flink-libraries==1.14.4
pyflink==1.0

I am facing an error regarding

from pyflink.connector.gcp.pubsub import PubSubSource

ModuleNotFoundError: No module named 'pyflink.connector'

. I want to use the pubsub connector to stream the data to SIEM servers. I came across this documentation https://nightlies.apache.org/flink/flink-docs-release-1.17/api/python/reference/pyflink.datastream/connectors.html and wanted to know that is there any available implementation for pubsub connector in flink for python?

You do notice that the python libs are not 1.17? Not sure if this affects it.

I have also updated the libraries

apache-flink==1.17.0

apache-flink-libraries==1.17.0

That didn't help.

Have you followed https://nightlies.apache.org/flink/flink-docs-stable/docs/dev/python/faq/#preparing-python-virtual-environment for installation instructions?

Thanks for responding, Actually I have application in which I want to integrate Flink and the app is deployed in docker containers. Is there any official guide for integrating the pyflink with my app which is running on dockerized environment? I added

apache-flink

in my requirements.txt and I am able to import pyflink.datastream but not the pyflink.connector.

You mean https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/resource-providers/standalone/docker/#using-flink-python-on-docker ?

yes I am able to run the docker container using the instruction but not sure how to include pubsub connector in the package.

@Bhavya Soni Hey, Pubsub is still not supported in PyFlink. I’m curious where the exception comes from.

Oh I see, So I will need to define the tasks to stream data from pubsub manually, right? thanks for the update @Dian Fu

@Bhavya Soni In PyFlink, connector is just a wrapper of the Java ones. So I think you could just write one and could refer to https://github.com/apache/flink/pull/20627 as an example.

Sure @Dian Fu, I was exploring the code and tried to run the same code on my dockerized environment. I have built the image from

flink:1.17.1

and I assume that all the necessary jar files will be available inside the container environment to build the connector/wrapper.

Connectors aren’t bundled with Flink

As I said I tried to add the jar files manually under

/dist-packages/pyflink/lib

directory. That also didn't solve the error.

I don’t know PyFlink, but in order to load connectors the JARs must be in the

lib

folder

The path I have provided is for lib directory under my python pyflink package. Is this where we add the JARs?

Wish I knew :)

I found this similar issue on stack-overflow https://stackoverflow.com/questions/69335471/typeerror-could-not-found-the-java-class-org-apache-flink-streaming-connectors and added the JARs at the given destination directory but didn't work.

@Bhavya Soni You need to add the transitive dependencies as well…

I got the jar file from here https://mvnrepository.com/artifact/org.apache.flink/flink-connector-gcp-pubsub