Hi, I'm very new to Flink and I'm trying to implement a online machine learning algorithm (a tree based one written in Python) pipeline with pyflink-ml, but I can't seem to find a detailed quickstart on how exactly to turn my algorithm into a python-based model that works with flinkml apis. My model takes an unbounded stream of text and adjusts itself for each data point, while producing a prediction text. I have two core questions after going through the codebase and docs, any help is appreciated: 1. What is model data in the flink-ml library? I'm assuming it to be the trained model weights, am I right? 2. I can only see java-based models in the codebase while Python simply uses the wrapped java models, is it true that we can implement Python-based models? (sorry I know this is a dumb question) 3. If 2 is feasible, will Python model get somehow translated to Java code under the hood or they could become a bottleneck in terms of processing?

Hey @Superskyyy I guess you could try to write a PyFlink job which reads data from unbounded stream and then performing transformations(predication) on the input data. You could look at how a PyFlink jobs look like: https://nightlies.apache.org/flink/flink-docs-release-1.16/docs/dev/python/datastream/intro_to_datastream_api/#common-structure-[…]ream-api-programs.

@Dian Fu Thank you for the link! Is there any specific reason or situation where I should use the Flink-ML lib instead of pure PyFlink? I'm trying to understand the core benefits of the ML lib. (My guess is the additional convenience of using open-sourced algorithms?)

My guess is the additional convenience of using open-sourced algorithms?

Yes, I think so.