Hello all, if Flink uses JDBC to write to TiDB are...
# randoms
Stan Hsu
07/19/2023, 2:25 AMHello all, if Flink uses JDBC to write to TiDB are multiple insert DDL in the one transaction? Because the follow-up Ticdc don't capture the changelog completely.
m
Martijn Visser
07/19/2023, 7:09 AMI don't think that the JDBC connector supports TiDB at this moment?
s
Stan Hsu
07/19/2023, 7:43 AMTiDB is highly compatible with the MySQL protocol, so can connect using JDBC.
m
Martijn Visser
07/19/2023, 7:44 AMBut still, from a Flink POV, there's never been any testing done to see if it works with TiDB
Martijn Visser
07/19/2023, 7:45 AMWhich API of Flink are you using?
s
Stan Hsu
07/19/2023, 7:46 AMI have successfully used JDBC to insert data to tidb, but there will be multiple inserts for the same primary key data in one transaction
Stan Hsu
07/19/2023, 7:46 AMI use flink sql jdbc connector
Stan Hsu
07/19/2023, 7:48 AMWhat I want to find out is whether it is the mechanism of flink itself or tidb. I am asking questions at the same time in the communities on both sides.
m
Martijn Visser
07/19/2023, 7:48 AMHow can there be multiple inserts for the same primary key from a Flink perspective?
s
Stan Hsu
07/19/2023, 7:49 AMm
Martijn Visser
07/19/2023, 7:49 AMYou have one incoming record/event and you're writing that to a JDBC sink. I'm assuming you have set up a primary key in your Flink table definition, so it operates in upsert mode, right?
s
Stan Hsu
07/19/2023, 7:49 AMyes
m
Martijn Visser
07/19/2023, 7:51 AMYes, so when you have one incoming record/event, you have a primary key defined in your table DDL and you're writing that to a MySQL database, Flink will use
INSERT .. ON DUPLICATE KEY UPDATE ..Martijn Visser
07/19/2023, 7:52 AMBut there won't be multiple insert ~DDLs~: DML every incoming record/event will be posted as
INSERT .. ON DUPLICATE KEY UPDATE ..s
Stan Hsu
07/19/2023, 7:53 AMone record one DDL?
m
Martijn Visser
07/19/2023, 7:53 AMSorry, meant DML, not DDL
s
Stan Hsu
07/19/2023, 7:54 AMyes, DML
Stan Hsu
07/19/2023, 7:56 AMI have set sink.buffer-flush.interval='0'/sink.buffer-flush.max-rows='0' to disable the batch sink, but there are still multiple DMLs in one transaction, which does not seem to be the mechanism of flink itself?
m
Martijn Visser
07/19/2023, 7:59 AMI'm under the impression that Flink will send multiple
INSERT .. ON DUPLICATE KEY UPDATE ..s
Stan Hsu
07/19/2023, 8:01 AMOk, thanks for your help, I will ask the tidb community.