Hi all 👋 — I've been writing a lot of Kafka/JDBC/HTTP connector tests for Flink jobs and kept rewriting the same Testcontainers wiring + polling-for-results boilerplate, so I put together a small JUnit 5 harness for it: flink-testkit (github.com/bharathgunapati/flink-testkit). Rough idea:

@KafkaTopic

/

@JdbcTable

/

@HttpEndpoint

on a typed static field gives you a handle for seeding data, awaiting outcomes (by count or predicate, no fixed sleeps), and getting the connection info to configure your job — one declaration instead of separate container/topic/consumer-group setup per test. It's early (v0.0.1, tested against Flink 1.19/1.20, HTTP examples on 2.2) and I'd genuinely like feedback before going further: - Is this solving a real pain point for people here, or do most of you already have this solved in-house? - Anything obviously missing for it to be useful (other connectors, other patterns)? - Any concerns with the approach itself? Not trying to replace Testcontainers or anyone's existing setup — just curious if this is useful to anyone outside my own projects. Any feedback would be appreciated.

my colleague wrote an article that folk here might find interesting: developer.confluent.io/blog/flink-oomkills

Hi everyone! 👋 A small milestone and a big 𝘁𝗵𝗮𝗻𝗸 𝘆𝗼𝘂 to the Flink community. Over the past several months, I've shared updates here about Str:::Lab Studio, and I wanted to say thanks for all the encouragement, discussions, and feedback along the way. The project has now crossed 10,000 Docker Hub downloads 🎉 I honestly don't know how many people are actively using it day-to-day, but seeing it approach 10K pulls tells me that people in the Flink community are at least checking it out, experimenting with it, and hopefully finding it useful. As an independent open-source project, that's incredibly motivating. The project has continued to evolve with improvements around the Flink SQL development experience, including pipeline visualization, watermark analysis, performance profiling, Schema Registry integration, and other quality-of-life features. I'd really appreciate hearing from anyone who has tried it: 1. What's working well? 2. What isn't? 3. What features would make it more useful in your Flink workflows? Even a few minutes of honest feedback would help me prioritize where to invest my time next. I'm also starting improvement work on one of the biggest roadmap items 𝗔𝗜 𝗔𝗴𝗲𝗻𝘁 𝗕𝘂𝗶𝗹𝗱𝗲𝗿 for streaming applications. If anyone in the community is interested in contributing (Java, React, Flink integrations, UX, documentation, or ideas), I'd be happy to collaborate. Thanks again to everyone who's downloaded the project, starred the repository, opened issues, or simply followed the progress. Community support is what keeps independent open-source projects moving forward. GitHub: github.com/coded-streams/strlabstudio Docker Hub: hub.docker.com/r/codedstreams/strlabstudio Looking forward to your feedback, and here's to the next milestone: 𝟭𝟬𝗞 𝗱𝗼𝘄𝗻𝗹𝗼𝗮𝗱𝘀!

Interesting Flink Links for July 2026: rmoff.net/2026/07/29/interesting-links-july-2026#…

Thanks for highlighting StreamFusion @rmoff! If anyone's curious about it over at Confluent, I'd love to chat further!

github.com/datafusion-contrib/StreamFusion Hi all, I just wanted to give a brief update on this project! 1. StreamFusion now supports the majority of SQL operators 2. StreamFusion achieves better performance than stock Flink in all local NexMark benchmark runs using exactly once semantics, with a kafka source/sink, and both in memory/local disk backends 3. After getting this project into a sufficient "alpha" state, I'll implement some more sink connectors (iceberg, paimon, etc) to try and get more adoption here Please let me know if you have any questions about the project or if you want to try it out/contribute, I'd be happy to chat!

Hi Everyone ! Currently working on having support for Glue Schema Registry on Flink SQL for Avro, Protobuf and JSON. I started a PR first for Avro, and already have the implementation for Protobuf and JSON stacked in my fork. Could anyone review? github.com/apache/flink-connector-aws/pull/236

Interesting Flink Links - August 2026 rmoff.net/2026/08/20/interesting-links-august-2026#…

My colleague Lorenzo Nicora just published two excellent posts all about Flink watermarks, check them out! Absolutely Everything You Always Wanted to Know About Watermarks 1️⃣ Part 1: Apache Flink 2️⃣ Part 2: Confluent Cloud for Apache Flink

A new blog post, from Flink committer @Gustavo de Morais: 🔗 Flink SQL Evolution: Handling Custom CDC with FROM_CHANGELOG and TO_CHANGELOG

Hello Flink community! 👋 I hope it's ok to share a small very-very-early stage project I'm building - Flink Terminal UI that is not far from parity with main Flink UI: github.com/nikitasavinov/flink-tui As a Flink enthusiast (and a person who worked with Flink for a few years on a daily basis) I was always missing this piece so decided to try doing it myself. Disclaimer: early alpha version, and I'm still looking for a good "feel" of the UX so expect breaking changes.

We published a blog post on our journey from in-house Flink autoscaler to the OSS community based Flink autoscaler at Netflix: netflixtechblog.com/a-tale-of-two-flink-autoscalers…

Hi All , what is starting point for deploying flink into ocp cluster, is there any starting document

github.com/datafusion-contrib/StreamFusion Wanted to flag that StreamFusion now supports an Apache Paimon source and sink! These are accelerated flink SQL jobs for writing to open table formats (paimon also has a mode to write iceberg metadata on top of its own tables!!). Feel free to star the repo if you're interested or reach out to me!

A Couple of great blog posts from @David Anderson: • Deduplicating Streams with Flink SQL • How I Review Flink SQL Solutions

Interesting Flink Links - September 2026 rmoff.net/2026/09/29/interesting-links-september-2026#…

👋 Hi everyone, I wrote about practical ways to cut data streaming costs in Kafka and Flink by optimizing batching, serialization, state, network traffic, and capacity. streamingdata.tech/p/improving-cost-efficiency-of-data-streaming-pipelines

Heya all.. I wrote about Flink x Deletion Vectors for those interested ipolyzos.substack.com/p/apache-flink-x-iceberg-deletion-vectors?r=anqog&…

Hi , if I have multiple jobs in operator mode there all and end up with the same context URL, so how can I configure context path for different different JOBS? So is there any reverse proxy available? We are using flink operator in openshift cluster It is community driven operator

Hi everyone, I’m migrating a Flink elastic search storage job from the ES7 connector to the ES8 connector and need guidance on failure handling. Setup: • Flink 2.2.1, Java 21. • Existing:

flink-connector-elasticsearch7:4.0.0-2.0

, client 7.10.2. • New:

flink-connector-elasticsearch8:4.0.0-2.0

, overriding the Java client to 8.19.22.

.setBulkResponseInspectorFactory(new ElasticsearchFailureHandler(host))
.setBulkFlushBackoffStrategy(FlushBackoffType.EXPONENTIAL, maxRetries, delay)

Our custom inspector classified failed bulk items and decided whether to ignore selected permanent document errors, perform restricted recovery, or fail the task. With

Elasticsearch8AsyncSinkBuilder

, I cannot find equivalent failure-handler or bounded-retry options. The released writer appears to requeue all failed bulk items without distinguishing permanent errors from transient ones or limiting attempts. Permanent failures could therefore retry indefinitely; task restarts can also replay the same problematic records. Is there a supported replacement/extension point for error classification, bounded retries/backoff, and durable DLQ handling, without maintaining a custom sink/writer? Was this intentionally changed with the async-sink architecture, or is it a feature-parity gap? I found FLINK-35546 and PR #107. Are these the right items to follow? We want to quarantine individual bad records or run time exceptions from elastic search such as index closed , shrink exceptions safely.

Hi, Do you have a stateful streaming Flink job that has big state? Did you ever had to wait hours for your job to start up just because you have added a new field to your state model? I certainly did ans started to ask myself a question: why I don't ever need to wait for all files to be rewritten in Data Lake just because I added new column? Why I am not rewriting all messages in Kafka just because some field was dropped? It is because that Flink hold only one schema at a time so it has to rewrite all state bytes when state schema changes. My solution: create a serializer that utilizes same approach as Confluent Avro Kafka serializer, that is to register schema in schema registry and store the schema id along with the data. Then when we deserialize bytes we read the data in old schema and project to current runtime schema. I would like to invite you to test the solution github.com/gliter/flink-schema-resolving-avro-type-serializer central.sonatype.com/artifact/io.github.gliter/schema-resolving-avro-serializer/overview There is even option that allows us to use this new serializer on top of an existing job that already uses Avro serializer. I have also a plan to embed schemas inside state serializer, so we don't need external schema registry.

🚀 Introducing Cobble-Flink — a unified storage layer for Apache Flink. Cobble-Flink brings together a state backend and SQL source/sink connectors, powered by the Cobble storage engine. It makes both streaming state and sink output available for fast sharing, querying, and reuse across jobs, while improving performance and preserving the familiar, native Flink experience. Highlights: • Performant stateful streaming — up to 2x throughput in Nexmark and 2.2x throughput in the Flink state benchmark, compared with RocksDB. • Disaggregated state — available starting from Flink 1.17, helping reduce resource usage and speed up restarts, with remote compaction to further offload CPU work from processing jobs. • Source and sink support — store intermediate results, and scan or look up persisted state and sink output from other Flink jobs. • Web Monitor — inspect records, explore state by column, and track changes across snapshots. • New: async state support for Flink 2.x 🎬 Here’s a short walkthrough: F0CB7285PME 📖 Docs and getting started: Just add the dependency to get started—make your data easier to work with, with high-performance storage for Flink. cobble-project.github.io/cobble-flink/latest These capabilities are powered by the Cobble storage core. We also welcome ideas for integrating it with other engines and applications: github.com/cobble-project/cobble The project is evolving quickly. Give it a try—we’d love to hear your feedback and suggestions!

Hey all, just wanted to share that StreamFusion (OSS flink accelerator on datafusion) now supports kafka, iceberg, paimon, and fluss sources and sinks! Feel free to check it out 🙂 github.com/datafusion-contrib/StreamFusion

Two new Flink SQL sink connectors — Qdrant and DuckDB Hi everyone, I’ve been working on two Flink SQL sink connectors recently, one for Qdrant and another for DuckDB, and wanted to share them here. The idea behind both is pretty simple: write streaming results directly into these systems using Flink SQL, without having to write a custom DataStream job just to handle the sink. Why these connectors? I couldn’t find an established Flink SQL sink connector for either of these use cases. • Qdrant: Useful for vector search and RAG pipelines, where embeddings generated in a streaming pipeline need to be written into a vector database. • DuckDB: A convenient option for local analytics, where it would be useful to write streaming results directly into a DuckDB database file. I wanted to see what it would take to make both work as regular Flink SQL connectors. A couple of interesting challenges The implementation turned out to be more interesting than simply writing records to a database. 1. Qdrant — a gRPC issue with shaded JARs Qdrant’s Java client uses gRPC. While packaging the connector into a shaded JAR, I ran into an issue with gRPC service-provider files under

META-INF/services

. Maven Shade’s default handling caused some of these files to be lost, which broke name resolution at runtime. The resulting error was quite misleading and took some digging to trace back to the packaging configuration. The fix was to add

ServicesResourceTransformer

to the Maven Shade configuration so that the service files are merged correctly. This isn't specific to Qdrant; it’s something worth watching out for when packaging connectors that depend on gRPC. I also added schema validation at table-creation time so unsupported types, such as

DECIMAL

,

TIMESTAMP

, and nested types, are rejected early with the offending column identified. 2. DuckDB — handling concurrent writes DuckDB's local database file uses process-level locking, which creates a problem when multiple Flink sink subtasks try to write to the same file. Rather than allowing the job to get stuck or encounter unpredictable write behavior, the connector validates the sink parallelism and rejects values greater than 1 for local file targets. For remote targets that support concurrent access, such as MotherDuck, higher parallelism can be supported. I also made the delivery behavior explicit: regular inserts provide at-least-once semantics, while upserts configured with

key-fields

use

ON CONFLICT

to make retries idempotent. Using the connectors Both connectors follow the usual Flink SQL workflow: define a table and use

INSERT INTO

. For example, with Qdrant:

CREATE TABLE embeddings (
    id STRING,
    text STRING,
    embedding ARRAY<FLOAT>
) WITH (
    'connector' = 'qdrant',
    'host' = 'localhost',
    'collection-name' = 'my_collection',
    'vector-field' = 'embedding',
    'id-field' = 'id'
);

INSERT INTO embeddings
SELECT id, text, embedding FROM source_stream;

And with DuckDB:

CREATE TABLE sink_table (
    id BIGINT,
    name STRING,
    score DOUBLE
) WITH (
    'connector' = 'duckdb',
    'database-path' = '/data/my.db',
    'table-name' = 't',
    'key-fields' = 'id'
);

INSERT INTO sink_table
SELECT id, name, score FROM source_stream;

Current status Both connectors target Flink 1.20 and are released under the Apache 2.0 license. They currently support the sink side only, and the target table or collection must already exist. • Qdrant: 13/13 tests passing, including an integration test using a real Qdrant Testcontainer. • DuckDB: 14/14 tests passing against an actual embedded database file, without requiring Docker. These are fresh projects, so there's still room to improve them based on real-world usage. Sharing them here in case anyone is working with Flink and Qdrant or DuckDB, or has run into similar issues with gRPC shading or concurrent writes to local database files. Feedback, suggestions, and issues are welcome! GitHub repositories: • Qdrant: github.com/Ashfaqbs/flink-connector-qdrant • DuckDB: github.com/Ashfaqbs/flink-connector-duckdb

Hey Bytewax folks! 👋 One week until OSA Con (Nov 2)! It’s a free, anti-hype day of open source analytics + AI, in San Francisco and livestreamed online. Also, some of the event sponsors are hiring, so it could be a nice opportunity to connect with teams and explore new opportunities if you're looking (or just curious!). 📍 Nov 2 · SF + online · FREE → luma.com/eis7567t?… Would be great to see some of this community there! 🙌

👋 Hello, team!

Hey everyone! 👋 I've been working on a small open-source project called Flink MCP Server. It allows AI assistants to inspect and diagnose Apache Flink clusters through the Model Context Protocol (MCP). The main focus is read-only monitoring, job diagnostics, checkpoints, backpressure analysis, and multi-cluster support across different deployment environments. It's still evolving, and I'd really appreciate feedback from people who work with Flink in real environments. GitHub: https://github.com/hakkiburakalks/flink-mcp-server If you have a chance to take a look, I'd love to hear what you think, what might be missing, or what could be improved. Thanks! 🙌

Interesting Flink Links: October 2026 rmoff.net/2026/10/29/interesting-links-october-2026#…

Hi team, we’re using

flink-connector-elasticsearch8:4.0.0-2.0

with Java client

8.19.22

and encountering FLINK-38054: checkpoint recovery fails with a Kryo

ClassNotFoundException

for

IndexOperation

, even after Elasticsearch becomes available again. The fix was merged in PR #159, but is not included in our released connector version. Could you publish a new connector release containing this fix and share the expected timeline? We can help validate a release candidate.