Hi everyone, I'm planning to introduce Flink MySQL...
# flink-cdc
k
Hi everyone, I'm planning to introduce Flink MySQL CDC 3.6 into our company’s data platform for a MySQL → Kafka pipeline using the DataStream API. Our DB team runs pt-online-schema-change with
sql_log_bin=0
config for performance reasons, particularly to avoid generating binlog events for the table copy. This setting also applies to the shadow table’s DDL and the final
RENAME TABLE
. Application connections continue writing DML to the binlog normally. As a result, the connector does not receive the schema changes or the table swap. After the rename, it continues decoding row events using the old schema. In our PoC, adding a column in the middle of a table caused subsequent values to be mapped to incorrect columns. The job remained running without reporting a deserialization error, so this appears to present a risk of silent data corruption. We understand that scan.parse.online.schema.changes.enabled=true supports pt-osc by interpreting shadow-table schema changes and the subsequent swap. However, those DDL events are absent from the binlog in our environment. Is there someone who struggled with the same problem and could advise on the following? 1. Is there a supported way to detect and safely handle this scenario without logging the entire pt-osc copy operation? (I’m already aware of the
scan.parse.online.schema.changes.enabled
option, but my understanding is that it cannot handle this case because
sql_log_bin=0
prevents the shadow table’s DDL and the final
RENAME TABLE
from being written to the binlog. Please correct me if I’m mistaken.) 2. Can the connector use
TABLE_MAP_EVENT
metadata, potentially with
binlog_row_metadata=FULL
, to detect schema mismatches and refresh the schema, or at least fail rather than emit incorrectly mapped records? 3. If external notifications provide the DDL statement and binlog positions captured immediately before and after the rename, is there a supported mechanism to reconcile schema history and replay the affected events safely? These positions bracket the rename but do not identify its exact boundary. We would prefer to avoid maintaining a custom connector fork or synchronously blocking DB schema changes on a CDC acknowledgment. Any recommended configuration, operational approach, or related issue would be appreciated.