Hi Just wondering if anyone had read (or wants to read) this article about comparing Flink and Spark Real-Time Mode (RTM). It makes an extremely bold claim that

Spark can now process events in milliseconds; up to 92% faster than Flink

. Very wishful thinking I think and I'm guessing there is a hint of bias given that the article was written by Databricks. I've been reading up on their process for comparing the two (I've never used Spark so I don't understand aspects of the code), but on the face of it, it looks like: • Flink Job (parallelism = 1) | Spark Job (parallelism = 40 - i think this is what

spark.sql.shuffle.partitions

does) • Flink Job (runs on AWS Managed Apache Flink 1.18 - limits optimisation) | Spark Job (runs on Databricks cluster - could be optimised) Doesn't seem like a fair fight to me (although I'm happy to be told otherwise if it is). Here is the article and Github repo for reference if anyone wants to have a look and share their thoughts. I'd love to hear them Article: https://www.databricks.com/blog/real-time-mode-ultra-low-latency-streaming-spark-apis-without-second-engine Github Repo: https://github.com/databricks-solutions/latency-benchmarks/tree/main

Hey, @marciano avihay has recently dealt with this bold, but not necessarily true, statement 🙂 https://www.linkedin.com/pulse/benchmarking-sparks-real-time-mode-oss-vs-flink-avichay-marciano-fdykf

great reads, thanks for sending those over 🙌