Hello folks! I’m using Flink CDC with MySQL on AWS RDS Aurora. For those using the same setup, have you found a way to run snapshots using the reader instance? If that’s not possible, how are you handling this limitation? Thanks!

you mean reading snapshot from Aurora read-only instance? In theory, you can skip backfill of snapshot and thus Flink CDC will only use the JDBC protocol to scan snapshot data of MySQL Table, but there’re some existing unnecessary command call like

show binary logs

which we need to remove it firstly. Could you open a JIRA ticket for this requirement?

Hey @Leonard Xu thanks for clarifying. I do have a ticket (FLINK-39169) for this, actually I was asking just in case anyone has solved this limitation. The proposal on the ticket is when starting a CDC job for this kind of architectures, run the snapshot on the reader and then switch to the writer for listening the binlog. I was imagining a spec like this:

source:
   type: mysql
   name: MySQL Source
   hostname: writer-instance-endpoint
   reader_hostname: reader-instance-endpoint
   port: 3306
   username: admin
   password: pass
   tables: adb.\.*, bdb.user_table_[0-9]+, [app|web].order_\.*
   server-id: 5401-5404

Adding a

reader_hostname (optional)

When this parameter is present it should run the snapshots against this hostname and for streaming using keep using the writer default one. Of course for backward compatibility if not preset everything keeps as it is. What do you think about this approach?

I understand your point, Luca. Reading snapshot data from a MySQL read-only instance while fetching more real-time changelog events from the MySQL master instance can reduce I/O pressure on the primary instance and improve data freshness after switching to the master. However, achieving exactly-once semantics under this setup is currently challenging. Here’s why: Flink CDC records the binlog position (denoted as

master-position-1

) at the moment each snapshot chunk (split by primary key, like

id

belongs to [200, 399]) finishes scanning. When switching to changelog reading, for primary keys belonging to that chunk’s range, it only consumes changelog events after

master-position-1

, thereby guaranteeing exactly-once delivery. This mechanism requires that both the historical snapshot scan and the recorded binlog position originate from the same MySQL instance—otherwise, binlog positions across instances may not be consistent or comparable. If at-least-once semantics is acceptable, your proposed architecture becomes much more feasible: we can first record the current binlog position on the master instance (

master-position-2

), then perform the full-table snapshot scan on the read-only replica, and finally start consuming changelog events from the master instance starting from

master-position-2

. This approach relaxes the strict consistency requirement between snapshot and binlog positions, trading off potential duplicates for improved scalability and reduced primary load.

Yes, is exactly as you describe. One important note from AWS Aurora / RDS is that only the writer instance exposes the binlog. The reader instances don’t expose it and it’s not possible to enable it there. So the writer effectively acts as the single source of truth for binlog events. Aurora uses storage replication, but from my understanding it can still lag slightly (in the order of milliseconds) so anyways it would be a-least-once semantics. I think is a reasonable trade-off if the goal is to reduce the load in the master instance. If you agree, I can go ahead with this and make it clear in the docs that exactly-once isn’t supported with this parameter. What do you think?

+1 from my side, Luca. Looking forward to your work.

Hello @Leonard Xu I just pushed the PR with the implementation. Whenever you can, I need a n approval to run the CI workflows, thanks!