Howto migrate a real-life batch pipeline from the DataSet API to the DataStream API Introduction # The Flink community has been deprecating the DataSet API since version 1.12 as part of the work on FLIP-131: Consolidate the user-facing Dataflow SDKs/APIs (and deprecate the DataSet API) . This blog article illustrates the migration of a real-life batch DataSet pipeline to a batch DataStream pipeline. All the code presented in this article is available in the tpcds-benchmark-flink repo. The use case shown here is extracted from a broader work comparing Flink performances of...

Apache Flink Kubernetes Operator 1.5.0 Release Announcement The Apache Flink community is excited to announce the release of Flink Kubernetes Operator 1.5.0! The release focuses on improvements to the job autoscaler that was introduced in the previous release and general operational hardening of the operator. We encourage you to download the release and share your feedback with the community through the Flink mailing lists or JIRA! We hope you like the new release and we’d be eager to learn about your experience with it.

Apache Flink 1.16.2 Release Announcement The Apache Flink Community is pleased to announce the second bug fix release of the Flink 1.16 series. This release includes 104 bug fixes, vulnerability fixes, and minor improvements for Flink 1.16. Below you will find a list of all bugfixes and improvements (excluding improvements to the build infrastructure and build stability). For a complete list of all changes see: JIRA. We highly recommend all users upgrade to Flink 1.16.2.

SIGMOD Systems Award for Apache Flink Apache Flink received the 2023 SIGMOD Systems Award, which is awarded to an individual or set of individuals to recognize the development of a software or hardware system whose technical contributions have had significant impact on the theory or practice of large-scale data management systems: The 2023 SIGMOD Systems Award goes to Apache Flink: “Apache Flink greatly expanded the use of stream data-processing.” Winning of SIGMOD Systems Award indicates the high recognition of Flink’s...

Announcing three new Apache Flink connectors, the new connector versioning strategy and externalization New connectors # We’re excited to announce that Apache Flink now supports three new connectors: Amazon DynamoDB, MongoDB and OpenSearch! The connectors are available for both the DataStream and Table/SQL APIs. Amazon DynamoDB - This connector includes a sink that provides at-least-once delivery guarantees. MongoDB connector - This connector includes a source and sink that provide at-least-once guarantees. OpenSearch sink - This connector includes a sink that provides at-least-once guarantees.

Apache Flink Kubernetes Operator 1.6.0 Release Announcement The Apache Flink community is excited to announce the release of Flink Kubernetes Operator 1.6.0! The release features a large number of improvements all across the operator. We encourage you to download the release and share your feedback with the community through the Flink mailing lists or JIRA! We hope you like the new release and we’d be eager to learn about your experience with it. Highlights # Improved and simplified rollback mechanism # Previously the rollback mechanism had some...

Stateful Functions 3.3.0 Release Announcement The Apache Flink community is excited to announce the release of Stateful Functions 3.3.0! Stateful Functions is a cross-platform stack for building Stateful Serverless applications, making it radically simpler to develop scalable, consistent, and elastic distributed applications. This new release upgrades the Flink runtime to 1.16.2. The binary distribution and source artifacts are now available on the updated Downloads page of the Flink website, and the most recent Java SDK, Python SDK,,...

Announcing the Release of Apache Flink 1.18 The Apache Flink PMC is pleased to announce the release of Apache Flink 1.18.0. As usual, we are looking at a packed release with a wide variety of improvements and new features. Overall, 174 people contributed to this release completing 18 FLIPS and 700+ issues. Thank you! Let’s dive into the highlights. Towards a Streaming Lakehouse # Flink SQL Improvements # Introduce Flink JDBC Driver For SQL Gateway # Flink 1.

Apache Flink Kubernetes Operator 1.6.1 Release Announcement The Apache Flink Community is pleased to announce the first bug fix release of the Flink Kubernetes Operator 1.6 series. The release contains fixes for several critical issues, and some doc improvements for the autoscaler. We highly recommend all users to upgrade to Flink Kubernetes Operator 1.6.1. Release Notes # Bug # [FLINK-32890] Correct HA patch check for zookeeper metadata store [FLINK-33011] Never accidentally delete HA metadata for last state deployments Documentation improvement #...

Apache Flink Kubernetes Operator 1.7.0 Release Announcement The Apache Flink community is excited to announce the release of Flink Kubernetes Operator 1.7.0! The release introduces a large number of improvements to the autoscaler, including a complete decoupling from Kubernetes to support more Flink environments in the future. It’s important to call out that the release explicitly drops support for Flink 1.13 and 1.14 as agreed by the community. We encourage you to download the release and share your feedback with the community through the Flink...

Apache Flink 1.17.2 Release Announcement The Apache Flink Community is pleased to announce the second bug fix release of the Flink 1.17 series. This release includes 82 bug fixes, vulnerability fixes, and minor improvements for Flink 1.17. Below you will find a list of all bugfixes and improvements (excluding improvements to the build infrastructure and build stability). For a complete list of all changes see: JIRA. We highly recommend all users upgrade to Flink 1.17.2.

Apache Flink 1.18.1 Release Announcement The Apache Flink Community is pleased to announce the first bug fix release of the Flink 1.18 series. This release includes 47 bug fixes, vulnerability fixes, and minor improvements for Flink 1.18. Below you will find a list of all bugfixes and improvements (excluding improvements to the build infrastructure and build stability). For a complete list of all changes see: JIRA. We highly recommend all users upgrade to Flink 1.18.1.

Announcing the Release of Apache Flink 1.19 The Apache Flink PMC is pleased to announce the release of Apache Flink 1.19.0. As usual, we are looking at a packed release with a wide variety of improvements and new features. Overall, 162 people contributed to this release completing 33 FLIPs and 600+ issues. Thank you! Let’s dive into the highlights. Flink SQL Improvements # Custom Parallelism for Table/SQL Sources # Now in Flink 1.19, you can set a custom parallelism for performance tuning via the scan.

📣The community just published a Special Thanks page to appreciate the companies and organizations who sponsored machines or services for helping the development of Apache Flink: https://flink.apache.org/what-is-flink/special-thanks/

Apache Flink Kubernetes Operator 1.8.0 Release Announcement The Apache Flink community is excited to announce the release of Flink Kubernetes Operator 1.8.0! The release includes many improvements to the operator core, the autoscaler, and introduces new features like TaskManager memory auto-tuning. We encourage you to download the release and share your experience with the community through the Flink mailing lists or JIRA! We’re looking forward to your feedback! Highlights # Flink Autotuning # We’re excited to announce our latest addition to the...

Apache Flink CDC 3.1.0 Release Announcement The Apache Flink community is excited to announce the release of Flink CDC 3.1.0! This is the first release after the community accepted the donation of Flink CDC as a sub-project of Apache Flink, with exciting new features such as transform and table merging. The eco-system of Flink CDC keeps expanding, including new Kafka and Paimon pipeline sinks and enhancement to existing connectors. We’d like to invite you to check out Flink CDC documentation and have a try on the quickstart tutorial to...

Apache Flink CDC 3.1.1 Release Announcement The Apache Flink Community is pleased to announce the first bug fix release of the Flink CDC 3.1 series. The release contains fixes for several critical issues and improves compatibilities with Apache Flink. Below you will find a list of all bugfixes and improvements (excluding improvements to the build infrastructure and build stability). For a complete list of all changes see: JIRA. We highly recommend all users to upgrade to Flink CDC 3.

Apache Flink Kubernetes Operator 1.9.0 Release Announcement The Apache Flink community is excited to announce the release of Flink Kubernetes Operator 1.9.0! The release includes many improvements to the autoscaler and standalone autoscaler, as well as memory optimizations to the operator. There was also a lot of progress made to translate documentation to Chinese. We encourage you to download the release and share your experience with the community through the Flink mailing lists or JIRA! We’re looking forward to your feedback!

Announcing the Release of Apache Flink 1.20 The Apache Flink PMC is pleased to announce the release of Apache Flink 1.20.0. As usual, we are looking at a packed release with a wide variety of improvements and new features. Overall, 142 people contributed to this release completing 13 FLIPs and 300+ issues. Thank you! Let’s dive into the highlights. Standing on the Eve of Apache Flink 2.0 # Flink 1.0 was released eight years ago. Since several months, the community is actively planning and taking steps towards the next major release.

Apache Flink CDC 3.2.0 Release Announcement The Apache Flink Community is excited to announce the release of Flink CDC 3.2.0! This release aims to improve usability and stability of existing features, including transform and schema evolution. Also, backwards-compatible code and tests have been added to help users upgrade from previous CDC versions more smoothly. Flink CDC release packages are available at Releases Page, and documentations are available at Flink CDC documentation page. Looking forward to any feedback from the community...

Preview Release of Apache Flink 2.0 The Apache Flink community is actively preparing Flink 2.0, the first major release since Flink 1.0 launched 8 years ago. As a significant milestone, Flink 2.0 is set to introduce numerous innovative features and improvements, along with some compatibility-breaking changes. To facilitate early adaptation to these changes for our users and partner projects (e.g., connectors), and to offer a sneak peek into the exciting new features while gathering feedback, we are now providing a preview...

Apache Flink Kubernetes Operator 1.10.0 Release Announcement The Apache Flink community is excited to announce the release of Flink Kubernetes Operator 1.10.0! The release includes several improvements to the autoscaler, and introduces a new Kubernetes custom resource called FlinkStateSnapshot to manage job snapshots. The process of job upgrades has also been enhanced which makes it possible to now use the last-state upgrade mode with session jobs. We encourage you to download the release and share your experience with the community through the Flink...

Introducing the new Amazon Kinesis Data Stream and Amazon DynamoDB Stream sources We are pleased to introduce updated versions of the Amazon Kinesis Data Stream and Amazon DynamoDB Stream sources. Built on the FLIP-27 source interface, these newer connectors introduce 7 new features and are compatible with Flink 2.0. The new KinesisStreamsSource replaces the legacy FlinkKinesisConsumer; and the new DynamoDbStreamsSource replaces the legacy FlinkDynamoDBStreamsConsumer. The new connectors are available for Flink 1.19 onwards, and AWS Connector version 5.0.0 onwards. For...

Apache Flink CDC 3.2.1 Release Announcement The Apache Flink Community is pleased to announce the first bug fix release of the Flink CDC 3.2 series. The release contains fixes for several critical issues and improves compatibilities with Apache Flink. Below you will find a list of all bugfixes and improvements (excluding improvements to the build infrastructure and build stability). For a complete list of all changes see: JIRA. We highly recommend all users to upgrade to Flink CDC 3.

Introducing the new Prometheus connector We are excited to announce a new sink connector that enables writing data to Prometheus (FLIP-312). This articles introduces the main features of the connector, and the reasoning behind design decisions. This connector allows writing data to Prometheus using the Remote-Write push interface, which lets you write time-series data to Prometheus at scale. Motivations for a Prometheus connector # Prometheus is an efficient time-series database optimized for building real-time dashboards and...

Apache Flink CDC 3.3.0 Release Announcement The Apache Flink Community is excited to announce the release of Flink CDC 3.3.0! This release introduces more features in transform and connectors and improve usability and stability of existing features. Flink CDC release packages are available at Releases Page, and documentations are available at Flink CDC documentation page. Looking forward to any feedback from the community through the Flink mailing lists or JIRA! Highlights # Connectors # New Pipeline Connectors # Flink CDC 3.

Apache Flink 1.20.1 Release Announcement The Apache Flink Community is pleased to announce the first bug fix release of the Flink 1.20 series. This release includes 75 bug fixes, vulnerability fixes, and minor improvements for Flink 1.20. Below you will find a list of all bugfixes and improvements (excluding improvements to the build infrastructure and build stability). For a complete list of all changes see: JIRA. We highly recommend all users upgrade to Flink 1.20.1.

Apache Flink 1.19.2 Release Announcement The Apache Flink Community is pleased to announce the second bug fix release of the Flink 1.19 series. This release includes 73 bug fixes, vulnerability fixes, and minor improvements for Flink 1.19. Below you will find a list of all bugfixes and improvements (excluding improvements to the build infrastructure and build stability). For a complete list of all changes see: JIRA. We highly recommend all users upgrade to Flink 1.19.2.

Apache Flink Kubernetes Operator 1.11.0 Release Announcement The Apache Flink community is excited to announce the release of Flink Kubernetes Operator 1.11.0! The version brings a number of important fixes and improvements to both core and autoscaler modules. We encourage you to download the release and share your experience with the community through the Flink mailing lists or JIRA! We’re looking forward to your feedback! Highlights # Flink 2.0 Preview Support # The Flink Kubernetes Operator and Autoscaler 1.

Apache Flink 2.0.0: A new Era of Real-Time Data Processing Today, the Flink PMC is proud to announce the official release of Apache Flink 2.0.0! This marks the first release in the Flink 2.x series and is the first major release since Flink 1.0 launched nine years ago. This version is the culmination of two years of meticulous preparation and collaboration, signifying a new chapter in the evolution of Flink. In this release, 165 contributors have come together to complete 25 FLIPs (Flink Improvement Proposals) and 369 issues.