How is pyflink going? 🙂 (85% of the workload in spark is pyspark in previous company I have seen, another 15% is ingestion)

It has already aligned most of the functionalities provided in the Java API, especially after the upcoming Flink 1.16 where the last pieces of unaligned work has been filled up, e.g. broadcast state, side out, window support, more built-in connector & format support, etc.

Thanks, @Dian Fu this is awesome

That's great to hear, @Dian Fu

thanks @Dian Fu. What about unit tests? I saw your repo and trying to adopt this approach, however, the test takes a long time (understandable since it needs to download jar and call JVM code etc..). Is there any plan to have similar unit tests capability as Java or at least include some of the test utils into the python distro package to avoid copy and paste code? Cheers

@Duc Anh Khu This is already supported in Flink 1.16. See https://issues.apache.org/jira/browse/FLINK-27421 for more details.