Hi all, running into an issue with table discovery. Setup: • Flink CDC: 3.6.0 • Flink: 1.20 • Pipeline: MySQL CDC source to StarRocks sink (YAML pipeline) Issue: On job start, the source correctly discovers and captures all tables that already exist and match my

tables:

pattern. However, if a new table is created in MySQL after the job has started (while it's actively streaming), it's never picked up automatically, even though it matches the same pattern and would've been captured had it existed at startup. Confirmed workaround: restarting the job fresh does pick up the new table correctly. So the discovery logic itself works fine; it's specifically the "table created while streaming, without a restart" case that doesn't. What I need: a way for the newly created table to be picked up automatically without restarting the job. I tried enabling

scan.newly-added-table.enabled

along with several

debezium.*

schema history passthrough options, but still couldn't get it to work without a restart:

source:
  type: mysql
  hostname: mysql
  port: 3306
  username: root
  password: rootpassword
  tables: cdc_demo.users,cdc_demo.users_new
  server-id: 5400-5404
  scan.startup.mode: initial
  jdbc.properties.allowPublicKeyRetrieval: "true"
  jdbc.properties.useSSL: "false"
  # Use flat dot notation for Debezium properties:
  debezium.database.history.store.only.captured.tables.ddl: "false"
  debezium.database.history.store.only.captured.databases.ddl: "false"
  debezium.schema.history.internal.store.only.captured.tables.ddl: "false"
  debezium.include.schema.changes: "true"
  scan.newly-added-table.enabled: "true"

sink:
  type: starrocks
  jdbc-url: jdbc:<mysql://starrocks-fe:9030>
  load-url: starrocks-fe:8030
  username: root
  password: ""
  sink.buffer-flush.interval-ms: 1000

pipeline:
  name: MySQL_to_StarRocks_CDC
  parallelism: 1
  schema.change.behavior: evolve

Question: Is live (no restart) discovery of newly created tables supported today in 3.6.0, and if so, what am I missing? If it's genuinely not supported, I'd like to raise a feature/bug ticket for it. Is this the right place to flag that first, or should I go straight to JIRA? Original config for reference (MySQL to StarRocks):

source:
  type: mysql
  tables: cdc_demo.users,cdc_demo.users_new
  scan.startup.mode: initial
  server-id: 5400-5404

Thanks in advance!

Hey! You want to remove

scan.newly-added-table.enabled

and add

scan.binlog.newly-added-table.enabled

, set it to

true

.

Hey @sap1ens! Thanks, it worked. But our actual use case is to handle gh-ost and pt-osc scenarios. I saw one more option below

newly-added-table

,

scan.parse.online.schema.changes.enabled

. I want to use this in production, but it says "experimental." I tested it and it's working fine. If I take this to production, what are the possible issues that could arise in the future due to it being experimental?

We’ve used the two options in production, maybe we can remove the experimental annotation in latest flink cdc version.

Can I do the documentation edit part?

sure, feel free to open an issue to improve this part.

issues.apache.org/jira/browse/FLINK-40532 Can someone please have a look and assign this to me?