Hey Flink, I have a few questions regarding the Flink and Kafka connectors and two-phase commits. • Q1. As we already know, the only number of Flink-Kafka consumers we can make is the total number of partitions in Kafka, so how does Flink scale horizontally? • Q2. I want to implement a two-phase commit with Kafka to Flink as the source and Flink to Postgresql as the sink. I want to create a POC for that; can anyone give me examples or documents? • Q3. I have my custom code written in GO for data transformation. To use Flink as ETL, I need to decide whether I need to convert my code to Java or whether I can directly implement GO in Flink.

On Q2 I don’t think the Flink Postgres connector supports 2PC

If I go towards Pinot or Clickhouse

There aren’t ASF Flink supported connectors for those

So than which DB I can use as a sink.

You can use all sorts of DBs, but DBs don’t support two phase commit sinks

Aah Okh got it so can you give me some flink or Kafka examples where 2PC is used in code.

There’s a variety of blog posts on this on the Flink website

Please let me know if you can answer the remaining two questions as well. Thanks

On Q1, you can make the number of consumers to be any number smaller or equal to number of partitions

Yes that is the problem the number of consumers will be equal to the number of Kafka partitions only so how can I horizontally scale then.

Why is that a problem?

One more thing how we can ensure exactly once consistency in DB.