This message was deleted.
# generals
Slackbot
09/28/2023, 4:08 PMThis message was deleted.
y
Yu Wei Sung
09/28/2023, 4:30 PMthe debezium pulsar source connectors (mysql, postgres, oracle,mongo …) basically wrap the debzium as embedded mode and db driver together and produce cdc events to pulsar topics.
m
Ming
09/28/2023, 5:36 PMTheir debezium server runs independently outside of Pulsar cluster, unlike Pulsar’s source connector which this is how Pulsar Debezium connectors are implemented).
Their debezium wasn’t multiple tenant. We pushed a fix to allow to use non-public/default (https://github.com/debezium/debezium/pull/4209).
q
Quanzheng Long
09/28/2023, 6:38 PMThanks both. Can you tell me a little more about how it works? Is it the code here? https://github.com/apache/pulsar/tree/master/pulsar-io/debezium
It seems like Pulsar connect of debezium is implemented as “embeded engine” of Debezium: https://debezium.io/documentation/reference/2.3/architecture.html#_embedded_engine
Is my understanding correct?
Thanks
y
Yu Wei Sung
09/28/2023, 6:53 PMyes, debezium relies on kafka as data transport. the pulsar source connector uses debezium embedded mode, pulsar’s kafka source connect (pickup kafka topic) and translate events to pulsar messages. for example,
postgres -> (jdbc -> debezium -> kafka -> pulsar kafka connector) -> pulsar topic.
q
Quanzheng Long
09/28/2023, 6:59 PMAwesome. I really appreciate your help!
Also, how does “Pulsar Debezium connector” recover from restarts? I assume it will resume from the last offset that succesfully publish to Pulsar topic
Quanzheng Long
09/28/2023, 7:02 PMDoes it just store the offset in a special topic? and then load it back (using latest) on restart?
m
Ming
09/28/2023, 8:08 PMI can speak for Postgres how it can recover from restarts. Postgres uses something called replication slot to keep track of WAL segments. Before Debezium reads the change, the segments in the slot won’t get deleted. This is implemented in Debezium. You are right that the connector records the position from which it has read changes in WAL. It is recorded and sends to the offset topic. Upon restarting of the connector, it reads the position from the offset topic. It’s the same implementation in Kafka Debezium.
🙏 1
✅ 1
q
Quanzheng Long
09/28/2023, 8:32 PMIs there any more documentation for configuring debezium for Puslar Debezium source?
For example, I want to use this filtering in Debezium, is that possible? https://debezium.io/documentation/reference/2.3/transformations/filtering.html
The config in the apache docs is quite limited: https://pulsar.apache.org/docs/3.1.x/io-cdc-debezium/#usage-1
Quanzheng Long
09/28/2023, 9:04 PMI tried to follow the code in https://github.com/apache/pulsar/tree/master/pulsar-io/debezium
It seems like using the debezium is not possible?
Seems like Pulsar debezium connect is NOT an debezium embeded engine? It’s just reusing the debezium source connector only…
Is that correct? or do I miss anything else.
Quanzheng Long
09/28/2023, 9:12 PMMy actual use case is that I want to limit what are the db tables are needed for CDC. I don’t want to send everything to the Pulsar .
Quanzheng Long
09/28/2023, 9:19 PMOh i see a config database.whitelist probably can do it.
Quanzheng Long
09/28/2023, 9:23 PMAhh no, this is only on database level.. I need to to make it table level..is that difficult to add that ?
Quanzheng Long
09/28/2023, 9:24 PMThe config said “there are other properties for listing databases and tables to include or exclude from monitoring.” I don’t see anywhere has the tables 😂
m
Ming
09/28/2023, 10:40 PMI use this link as a reference for my source connector config https://debezium.io/documentation/reference/stable/connectors/postgresql.html
Underneath it’s Kafka connector
2 Views