Title

Question

How can I resolve the 'The table is full' error when trying to replicate data from MySQL to BigQuery?

Answer

One possibility is to clear the temporary tables and try to re-run the sync. Alternatively, you may need to increase the allocated memory for Aurora. Upgrading the connector to the latest version may also resolve the issue. For improving performance in loading big tables in MySQL, create a separate thread specifically for that issue on the same channel.

m

Maykon Lopes

10/05/2022, 12:12 AM
[AWS Aurora MySQL /rdsdbdata/tmp is full]
Hey folks, I'm facing an issue to replicate data from a mysql to bigquery, the fail is really quickly in log output, showing this:
Caused by: java.lang.RuntimeException: java.sql.SQLException: The table '/rdsdbdata/tmp/#sql129d_35_1' is full
	at io.airbyte.db.jdbc.StreamingJdbcDatabase.unsafeQuery(StreamingJdbcDatabase.java:79)
	at io.airbyte.db.jdbc.JdbcDatabase.unsafeQuery(JdbcDatabase.java:195)
	at io.airbyte.integrations.source.relationaldb.AbstractRelationalDbSource.lambda$queryTable$0(AbstractRelationalDbSource.java:61)
	... 34 more
Caused by: java.sql.SQLException: The table '/rdsdbdata/tmp/#sql129d_35_1' is full
	at com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:129)
	at com.mysql.cj.jdbc.exceptions.SQLExceptionsMapping.translateException(SQLExceptionsMapping.java:122)
I tried increase size in aurora rds mysql parameter group, but still having the issue. Any clue?
s

Saj Dider (Airbyte)

10/05/2022, 2:39 PM
Hey @Maykon Lopes, how much memory did you allocate for the aurora and what is the size of the sync in terms of records and bytes? My guess is that you can clear the tmp tables and try to re-run the sync. Otherwise, you may not have increased the memory enough
m

Maykon Lopes

10/19/2022, 5:57 PM
Hey @Saj Dider (Airbyte) sorry my late answer here, the issue was solved after upgrade and connector to lastest version. Thanks for you help.
Now I'm struggling to improve performance in load big tables in mysql, tables with more than 300GB.
s

Saj Dider (Airbyte)

10/19/2022, 7:17 PM
Hey @Maykon Lopes, happy to hear that upgrading the connector fixed your issue. Create a separate thread for the performance issue with MySQL on this channel.