Hello <#C03G7LJTS2G|> can we do all the things usi...
# troubleshootings
Siddhesh Kalgaonkar
02/09/2023, 6:18 PMHello #C03G7LJTS2G can we do all the things using Python datastream API like Java datastream API
Which one is full fledged one?
m
Martijn Visser
02/09/2023, 6:22 PMJava is the full fledged one
Martijn Visser
02/09/2023, 6:22 PMPython uses the Java API under the hood
d
David Anderson
02/09/2023, 6:32 PMFWIW, since 1.16 the Python API is pretty complete. Async i/o is perhaps the most prominent feature that is still missing from the Python DataStream API.
s
Siddhesh Kalgaonkar
02/09/2023, 7:15 PMThanks a lot for your suggestions
I have a pipeline something like below
Trigger Kafka source > generate records > dump in a single window (parallelism is 1) here > aggregation kinda algorithm > dump records in Kafka topics
I think to achieve this Python datastream api would be enough since I don’t think so I would be dealing with Async I/O here but any suggestions would be helpful
TIA #C03G7LJTS2G
d
Dian Fu
02/10/2023, 1:59 AM@Siddhesh Kalgaonkar Yes, I think you can achieve this with Python DataStream API. Here are a few examples/documentations you could refer to:
https://github.com/apache/flink/tree/master/flink-python/pyflink/examples/datastream/windowing
https://github.com/apache/flink/tree/master/flink-python/pyflink/examples/datastream/connectors
https://nightlies.apache.org/flink/flink-docs-release-1.16/docs/dev/python/datastream/intro_to_datastream_api/
flink 1
s
Siddhesh Kalgaonkar
02/10/2023, 6:55 AMThanks a lot @Dian Fu I just have one more doubt when would Async operations be helpful?One example could be while inserting records in the db but apart from this can you give me couple of examples or any links that would help to analyse my use case better.
d
Dian Fu
02/10/2023, 8:44 AM@Siddhesh Kalgaonkar Async Operator is useful when you need to access external systems. For example, performing inference by accessing an external inference server, accessing KV systems(HBase, etc) to retrieve some values for each input, etc.
flink 1
s
Siddhesh Kalgaonkar
02/10/2023, 9:29 AMThanks for the insights but right now we don’t have a scope for any DBs.
As I mentioned about my current use case scope do you think we need Java for that ? Or apart from dbs will Async operator needed for Kafka connector?
Any links would also help to understand
Thanks a lot for your help though @Dian Fu
m
Martijn Visser
02/10/2023, 9:31 AMIf you have no scope for any DBs, you can use PyFlink and there's no need for using Java.
flink 1
s
Siddhesh Kalgaonkar
02/10/2023, 9:35 AMCool. Thanks a lot @Martijn Visser
Siddhesh Kalgaonkar
02/12/2023, 10:31 AMJust a question for Python do we have to use Mock and MagicMock/ Pytest like we do in the Python projects way for testing Flink programs or is there any other way of testing it ? Just need your guidance here since I didn't find any information on the documentation https://nightlies.apache.org/flink/flink-docs-release-1.16/docs/dev/datastream/testing/
m
Martijn Visser
02/12/2023, 11:47 AMPlease follow the rules and don’t @ people directly
s
Siddhesh Kalgaonkar
02/12/2023, 12:54 PMMy bad
Siddhesh Kalgaonkar
02/12/2023, 12:58 PMI have added a new post for it. Thanks for the guidance. I wasn’t aware about the channel rules :)
d
Dian Fu
02/13/2023, 2:57 AMI think you can following how we write tests for PyFlink, see https://github.com/apache/flink/tree/master/flink-python/pyflink/table/tests and https://github.com/apache/flink/tree/master/flink-python/pyflink/datastream/tests for more details.
Dian Fu
02/13/2023, 2:58 AMBesides, here is another example showing how to add tests in an external project: https://github.com/dianfu/pyflink-faq/tree/main/testing
s
Siddhesh Kalgaonkar
02/13/2023, 9:44 AMCool. I will check out these links @Dian Fu. Thanks alot for your help 🙂
2 Views