Two new Flink SQL sink connectors — Qdrant and DuckDB Hi everyone, I’ve been working on two Flink SQL sink connectors recently, one for Qdrant and another for DuckDB, and wanted to share them here. The idea behind both is pretty simple: write streaming results directly into these systems using Flink SQL, without having to write a custom DataStream job just to handle the sink. Why these connectors? I couldn’t find an established Flink SQL sink connector for either of these use cases. • Qdrant: Useful for vector search and RAG pipelines, where embeddings generated in a streaming pipeline need to be written into a vector database. • DuckDB: A convenient option for local analytics, where it would be useful to write streaming results directly into a DuckDB database file. I wanted to see what it would take to make both work as regular Flink SQL connectors. A couple of interesting challenges The implementation turned out to be more interesting than simply writing records to a database. 1. Qdrant — a gRPC issue with shaded JARs Qdrant’s Java client uses gRPC. While packaging the connector into a shaded JAR, I ran into an issue with gRPC service-provider files under

META-INF/services

. Maven Shade’s default handling caused some of these files to be lost, which broke name resolution at runtime. The resulting error was quite misleading and took some digging to trace back to the packaging configuration. The fix was to add

ServicesResourceTransformer

to the Maven Shade configuration so that the service files are merged correctly. This isn't specific to Qdrant; it’s something worth watching out for when packaging connectors that depend on gRPC. I also added schema validation at table-creation time so unsupported types, such as

DECIMAL

,

TIMESTAMP

, and nested types, are rejected early with the offending column identified. 2. DuckDB — handling concurrent writes DuckDB's local database file uses process-level locking, which creates a problem when multiple Flink sink subtasks try to write to the same file. Rather than allowing the job to get stuck or encounter unpredictable write behavior, the connector validates the sink parallelism and rejects values greater than 1 for local file targets. For remote targets that support concurrent access, such as MotherDuck, higher parallelism can be supported. I also made the delivery behavior explicit: regular inserts provide at-least-once semantics, while upserts configured with

key-fields

use

ON CONFLICT

to make retries idempotent. Using the connectors Both connectors follow the usual Flink SQL workflow: define a table and use

INSERT INTO

. For example, with Qdrant:

CREATE TABLE embeddings (
    id STRING,
    text STRING,
    embedding ARRAY<FLOAT>
) WITH (
    'connector' = 'qdrant',
    'host' = 'localhost',
    'collection-name' = 'my_collection',
    'vector-field' = 'embedding',
    'id-field' = 'id'
);

INSERT INTO embeddings
SELECT id, text, embedding FROM source_stream;

And with DuckDB:

CREATE TABLE sink_table (
    id BIGINT,
    name STRING,
    score DOUBLE
) WITH (
    'connector' = 'duckdb',
    'database-path' = '/data/my.db',
    'table-name' = 't',
    'key-fields' = 'id'
);

INSERT INTO sink_table
SELECT id, name, score FROM source_stream;

Current status Both connectors target Flink 1.20 and are released under the Apache 2.0 license. They currently support the sink side only, and the target table or collection must already exist. • Qdrant: 13/13 tests passing, including an integration test using a real Qdrant Testcontainer. • DuckDB: 14/14 tests passing against an actual embedded database file, without requiring Docker. These are fresh projects, so there's still room to improve them based on real-world usage. Sharing them here in case anyone is working with Flink and Qdrant or DuckDB, or has run into similar issues with gRPC shading or concurrent writes to local database files. Feedback, suggestions, and issues are welcome! GitHub repositories: • Qdrant: github.com/Ashfaqbs/flink-connector-qdrant • DuckDB: github.com/Ashfaqbs/flink-connector-duckdb