Hello all, We are currently having problems with a mysql source that keeps breaking when importing large data sets. We keep getting the same error from the log:

....
2022-11-20 10:21:55 [44msource[0m > Nov 20, 2022 10:21:55 AM com.github.shyiko.mysql.binlog.BinaryLogClient$5 run
2022-11-20 10:21:55 [44msource[0m > INFO: Keepalive: Trying to restore lost connection to <http://database-server.com:3306|database-server.com:3306>
2022-11-20 10:25:16 [44msource[0m > Stopping the task and engine
2022-11-20 10:25:16 [44msource[0m > Stopping down connector
2022-11-20 10:26:46 [44msource[0m > Coordinator didn't stop in the expected time, shutting down executor now
2022-11-20 10:28:16 [44msource[0m > Connection gracefully closed
2022-11-20 10:28:16 [44msource[0m > Stopped FileOffsetBackingStore
2022-11-20 10:28:16 [44msource[0m > Debezium engine shutdown.
2022-11-20 10:28:16 [44msource[0m > The main thread is exiting while children non-daemon threads from a connector are still active.
Ideally, this situation should not happen...
Please check with maintainers if the connector or library code should safely clean up its threads before quitting instead.
The main thread is: main (RUNNABLE)
 Thread stacktrace: java.base/java.lang.Thread.getStackTrace(Thread.java:1610)
        at io.airbyte.integrations.base.IntegrationRunner.dumpThread(IntegrationRunner.java:334)
        at io.airbyte.integrations.base.IntegrationRunner.watchForOrphanThreads(IntegrationRunner.java:282)
        at io.airbyte.integrations.base.IntegrationRunner.produceMessages(IntegrationRunner.java:219)
        at io.airbyte.integrations.base.IntegrationRunner.runInternal(IntegrationRunner.java:141)
        at io.airbyte.integrations.base.IntegrationRunner.run(IntegrationRunner.java:100)
        at io.airbyte.integrations.source.mysql.MySqlSource.main(MySqlSource.java:309)
2022-11-20 10:28:16 [44msource[0m > Active non-daemon thread: debezium-mysqlconnector-database-change-event-source-coordinator (TIMED_WAITING)
 Thread stacktrace: java.base@17.0.4.1/jdk.internal.misc.Unsafe.park(Native Method)
....

We can already rule out that it is an out of memory problem on our airbyte server. At least according to the server's logs, there seems to be no problem here. Do any of you have an idea how we could solve this problem?

Hey @Michael Sonnleitner, are you still experiencing this issue? There were some changes to the MySQL source connector adding more available memory to the JDBC buffer, can you try updating source-mysql to version 1.0.14 and see if that helps? If it continues, can you please attach the full sync log and tag me?

Hey @Sunny Hashmi (Airbyte), thank you for your reply. I tried it yesterday again with version 1.0.15. Unfortunately, the sync did not work again. Here are the complete, anonymised logs from the first attempt.

Hi Michael, unfortunately have not been able to reproduce this on our end. Do you have any monitoring on the mysql database, can you see if it's low on resources? I see at 161931 a keepalive thread was successfully started:

2022-12-07 16:19:31 [44msource[0m > Waiting for keepalive thread to start
2022-12-07 16:19:31 [44msource[0m > Creating thread debezium-mysqlconnector-source-database-binlog-client
2022-12-07 16:19:31 [44msource[0m > Skipping previously processed row event: Event{header=EventHeaderV4{timestamp=1670398593000, eventType=EXT_WRITE_ROWS, serverId=10, headerLength=19, dataLength=55, nextPosition=640513457, flags=0}, data=WriteRowsEventData{tableId=86, includedColumns={0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10}, rows=[
    [90934143, 658030473, 0, 2022-12-07T08:36:33, 1, 0.00, null, null, null, null, null]
]}}
2022-12-07 16:19:31 [44msource[0m > Keepalive thread is running

but then as you noticed, after

BinaryLogClient$5 run

at 162831, the keepalive somehow lost the connection:

2022-12-07 16:28:31 [44msource[0m > Dec 07, 2022 4:28:31 PM com.github.shyiko.mysql.binlog.BinaryLogClient$5 run
2022-12-07 16:28:31 [44msource[0m > INFO: Keepalive: Trying to restore lost connection to <http://server-source-database.com:3306|server-source-database.com:3306>
2022-12-07 16:32:19 [44msource[0m > Stopping the task and engine
2022-12-07 16:32:19 [44msource[0m > Stopping down connector
2022-12-07 16:33:49 [44msource[0m > Coordinator didn't stop in the expected time, shutting down executor now
2022-12-07 16:35:19 [44msource[0m > Connection gracefully closed
2022-12-07 16:35:19 [44msource[0m > Stopped FileOffsetBackingStore
2022-12-07 16:35:19 [44msource[0m > Debezium engine shutdown.

My understanding is that keepalive implies that the connection will not hit mysql timeouts, but it may be worth checking those as well if you can