for my paimon catalog i'm using a jdbc catalog interface backed by postgresql. I have a source table, which is postgresql based, as a cdc source, that I want to record into a catalog, what catalog are you guys using, don't want to use HMS, trying to simplify the stack.

to expand. my Flink sql, this would of course be applicable to any external source table (PostgreSQL, MySQL, MongoDB, REDIS ... etc).

-- Inbound from PostgreSQL via CDC Process
CREATE CATALOG postgres_catalog WITH 
    ('type'='generic_in_memory'); 

CREATE DATABASE IF NOT EXISTS postgres_catalog.demog;

In which I then create the below. I'd like to rather use a better catalog option, that includes persistence.

CREATE OR REPLACE TABLE postgres_catalog.demog.accountholders (
     _id                BIGINT                  NOT NULL
    ,nationalid         VARCHAR(16)             NOT NULL
    ,firstname          VARCHAR(100)
    ,lastname           VARCHAR(100)
    ,dob                VARCHAR(10) 
    ,gender             VARCHAR(10)
    ,children           INT
    ,address            STRING
    ,accounts           STRING
    ,emailaddress       VARCHAR(100)
    ,mobilephonenumber  VARCHAR(20)
    ,created_at         TIMESTAMP_LTZ(3)
    ,WATERMARK          FOR created_at AS created_at - INTERVAL '15' SECOND
    ,PRIMARY KEY (_id) NOT ENFORCED
) WITH (
     'connector'                           = 'postgres-cdc'
    ,'hostname'                            = 'postgrescdc'
    ,'port'                                = '5432'
    ,'username'                            = 'dbadmin'
    ,'password'                            = 'dbpassword'
    ,'database-name'                       = 'demog'
    ,'schema-name'                         = 'public'
    ,'table-name'                          = 'accountholders'
    ,'slot.name'                           = 'accountholders0'
    -- experimental feature: incremental snapshot (default off)
    ,'scan.incremental.snapshot.enabled'   = 'true'               -- experimental feature: incremental snapshot (default off)
    ,'scan.startup.mode'                   = 'initial'            -- <https://nightlies.apache.org/flink/flink-cdc-docs-release-3.1/docs/connectors/flink-sources/postgres-cdc/#startup-reading-position>     ,'decoding.plugin.name'                = 'pgoutput'
    ,'decoding.plugin.name'                = 'pgoutput'
);

If you’re evaluating systems for large-scale JSON analytics, such as Apache Doris(https://doris.apache.org/), Snowflake, ClickHouse, Elasticsearch, or more, start with this list of questions: 1️⃣ Can it handle 1K, 5K, 10K JSON fields without dropping performance? 2️⃣ Does it support type evolution without breaking ingestion? 3️⃣ Can you apply indexes selectively to the fields that matter? 4️⃣ Does query performance degrade after schema expansion? Apache Doris check these boxes ✅, thanks to the VARIANT data type. With Apache Doris VARIANT data type, you can: - Store raw JSON as a VARIANT column - Automatically extract JSON fields into sub-columns during ingestion - Define schema templates to control data types for groups of fields - Add indexes (inverted, bloomfilter, minmax, vector) to specific JSON patterns - Scale to 1K–10K+ JSON fields while keeping queries fast 🔗 Watch the full webinar for more info on Variant in Apache Doris (contains demo: querying 1 billion rows of JSON data in seconds): https://lnkd.in/g6JKcx4w

Hello! I have a question. 🙂 I have a question about JIRA registration. I requested to join the Flink JIRA some time ago with the email, but I haven't received a confirmation yet. Does anyone know where I could follow up on this request?

Hello guys! Could someone advise which channel is appropriate for raising a Flink issue? I'd like to start a thread but want to make sure it's the right place.

Hi there, could someone share a link on Paimon Slack channel if any?

ok, just going to ask as i'm loosing my cool/marbles/sanity here, trying to get this jar "pie" build. anyone have a docker compose for 1.20.1 or 1.20.2 that includes the following. iceberg 1.9.1 catalog on jdbc -> postgres, chosen as it can handle both apace iceberg and apache paimon Iceberg storage on Minion/S3 file format will be parquet. flink cdc 3.5.0

⏰ Last Call: Don’t miss our webinar on Hybrid Search & Context Engineering with Apache Doris 4.0 🎟️ RSVP: https://lnkd.in/gvczv4zi AI models won't work well without the right context. But retrieving the right context at scale (think datasets in the billions) is a real challenge. Join us to learn how Apache Doris 4.0 solves this with Hybrid Search, bringing together: 1️⃣ Vector search for semantic relevance 2️⃣ Full-text search (BM25 + inverted index) 3️⃣ High-performance structured analytics All in one unified SQL engine Come and see real use cases of how data teams are using Doris 4.0 to build accurate and cost efficient their AI application.

Got it working Busy writing up the docs. Will share as soon as published Apache Flink, with Apache iceberg, using a jdbc based catalog, with postgresql for catalog persistence. Object storage using MinIO. And Apache paimon, also using jdbc for catalog.

guys, how / where do we log a feature request, other projects allow you to raise a issue, and then tag it as a feature request.

Apache Polaris Catalog (includes the Polaris tools Console/UI) for Apache Iceberg, with Apache Flink in the middle... Storage on MinIO. https://medium.com/@georgelza/apache-iceberg-with-apache-polaris-as-catalog-with-p[…]istence-minio-object-store-dd7003d99723?postPublishedType=repub

guys, are there a UI for the JDBC based catalogs, that expose the catalogs, databases and tables and then the associated metadata etc down the stack...

happy happy new year guys. wishing you all an amazing 2026

🔥*Open Lakehouse & AI meetup in Chicago on Jan 22* Folks interested and/or using ClickHouse®, Apache Iceberg, lakehouse performance, or AI agents, this event is for you. Talks include: • Building a Foundation for AI with ClickHouse® + Iceberg — compute/storage separation, real perf results, roadmap, and how to get started (Robert Hodges, Altinity) • High-Performance Analytics on Iceberg — real bottlenecks and how modern engines fix them at scale (Chelsea Wang, CelerData) • Iceberg for Agents — turning lakehouse data into AI-ready context, structured RAG, and a live open-source demo (Andrew Madson, Fivetran) Expect real code and architecture and demos. 👉 https://luma.com/699ffm7v

I've been getting into Claude lately and didn't find any Flink skills available: https://skills.sh/?q=flink anyone thought making one? I might when I have some capacity but lack of anything available was a surprise. On another hand, I found Claude doing really well on FlinkSQL tasks but it did run into some repeatable issues that could have been avoided. If you don't know what skills are in Claude (and probably other coding agents as well) - lazily loaded, procedural instructions on how to achieve given goals. For reference: the most popular Kafka skill: https://skills.sh/404kidwiz/claude-supercode-skills/kafka-engineer

@Sreeja Govardhana joined #random. They’re also new to Apache Flink.

@Mohamed Motaz joined #random. They’re also new to Apache Flink.

@Munindra Naik joined #random. They’re also new to Apache Flink.

@Janakan Siva joined #random. They’re also new to Apache Flink.

Question, who is in charge of building a released version of flink-connector-mongodb ...need that awesome work Rahul Teke did to contribute the Flink 2 update

@Shaun Wanford has left the channel

@Jared Yu joined #random. They’re also new to Apache Flink.

@Ale joined #random. They’re also new to Apache Flink.

Hello team! I am looking around on flink project, to start make a contributions. I am new on apache organization I register for ICLA already but my request to jira board is pending apache by email. Is there someone, that I can contact , in order to see if everything is ok? Thanks!

@Hao Li joined #random. They’re also new to Apache Flink.

@김상진 joined #random. They’re also new to Apache Flink.

@Allyson Martins joined #random. They’re also new to Apache Flink.

Hi Just wondering if anyone had read (or wants to read) this article about comparing Flink and Spark Real-Time Mode (RTM). It makes an extremely bold claim that

Spark can now process events in milliseconds; up to 92% faster than Flink

. Very wishful thinking I think and I'm guessing there is a hint of bias given that the article was written by Databricks. I've been reading up on their process for comparing the two (I've never used Spark so I don't understand aspects of the code), but on the face of it, it looks like: • Flink Job (parallelism = 1) | Spark Job (parallelism = 40 - i think this is what

spark.sql.shuffle.partitions

does) • Flink Job (runs on AWS Managed Apache Flink 1.18 - limits optimisation) | Spark Job (runs on Databricks cluster - could be optimised) Doesn't seem like a fair fight to me (although I'm happy to be told otherwise if it is). Here is the article and Github repo for reference if anyone wants to have a look and share their thoughts. I'd love to hear them Article: https://www.databricks.com/blog/real-time-mode-ultra-low-latency-streaming-spark-apis-without-second-engine Github Repo: https://github.com/databricks-solutions/latency-benchmarks/tree/main