Hi everybody I want to consider using Apache Flink CDC that Apache Flink #troubleshooting

Hi everybody. I want to consider using Apache Fli...

Marco Villalobos

12/12/2025, 7:05 PM

Hi everybody. I want to consider using Apache Flink CDC that has Mysql as a Source and Paimon as a Sink. I wanted to deploy this on the Kubernetes Operator, but I noticed that it requires the Mini Cluster. I prefer not to use Mini Cluster for production. How do I change that? My next option is Amazon Managed Service for Apache Flink, but I don't know if people have successfully integrated that with Apache Flink CDC. Is there a way to change Apache Flink CDC to NOT use MiniCluster in the Kubernetes Operator? Does Apache Flink CDC work with Amazon Managed Service for Apache Flink? My alternative is integrate Debezium directly, with Apache Kafka, and Spark structred streaming with Paimon. I'd prefer to use Flink.

Alexey

12/15/2025, 3:59 PM

Hi, what do you mean by Mini Cluster? Flink App Cluster ?

Marco Villalobos

12/15/2025, 7:04 PM

AI Answer to first question: The Flink MiniCluster is a lightweight, embedded Apache Flink cluster that runs within a single Java Virtual Machine (JVM). It is primarily designed for local execution of Flink jobs, making it an essential tool for integration testing, debugging, and small-scale data processing tasks. I would prefer to deploy this as a normal horizontally scalable job within the Flink Kubernetes Operator.

Alexey

12/15/2025, 7:33 PM

it is possible

Marco Villalobos

12/15/2025, 7:59 PM

1. I don't see any instruction on how to achieve that. I see this in the Kubernetes section: Flink CDC submits a job to a remote Flink cluster by default, you should start a Standalone Flink cluster in the pod by

--use-mini-cluster

in Operator mode.

Marco Villalobos

12/15/2025, 8:00 PM

Do you know where I can find an example of how to deploy Flink CDC to the Flink Kubernetes Operator without a mini-cluster?

Marco Villalobos

12/15/2025, 9:13 PM

The only documentation I see is this: https://nightlies.apache.org/flink/flink-cdc-docs-master/docs/deployment/kubernetes/#create-a-flinkdeployment-yaml

Alexey

12/16/2025, 9:05 AM

I see what you mean. This is an example to run Flink CDC using Pipeline framework. No idea why MiniCluster is involved here. I would like to know as well. I would also try a JAR-based FlinkDeployment CR where inside my code I would construct Flink CDC pipeline. This should work for normal Flink Source from here: https://nightlies.apache.org/flink/flink-cdc-docs-master/docs/connectors/flink-sources/overview/

Alexey

12/16/2025, 9:08 AM

Copy code

- '--use-mini-cluster'
      - /opt/flink/flink-cdc-3.6-SNAPSHOT/conf/mysql-to-doris.yaml
    entryClass: org.apache.flink.cdc.cli.CliFrontend

I think this is just a way to launch CDC Pipeline via the CDC CLI, however I would expect it will use K8s as usually (JM + N TMs Pods)

Marco Villalobos

12/16/2025, 11:49 PM

but your example uses mini-cluster. Do you see the command line option your'e passing?

use-mini-cluster

. That is what I hope to avoid.

Alexey

12/17/2025, 6:59 AM

Have you tried to launch such an example?

Marco Villalobos

12/17/2025, 5:22 PM

I’m still working on the configuration, but I’m pretty close

Open in Slack

Previous Next