Channels
advice-data-architecture
advice-data-ingestion
advice-data-orchestration
advice-data-privacy
advice-data-quality
advice-data-transformation
advice-data-visualization
advice-data-warehouses
advice-reverse-etl
airbyte-cloud
airbyte-for-power-users
airbyte-help
airbyte-plus-airflow
airbyte-plus-dagster
announcements
authentication
careers
cloud-master-build-failure
community-strategy
connector-builds
connector-development
contributing-to-airbyte
databricks-airbyte316
databricks-airbyte316
delete-remove
dev
dev-3-alerts
dev-db-sources
dev-frontend
dev-frontend6323
dev1-alerts
events-and-conferences
ext-torch-cube
feedback-and-requests
github-updates
good-reads
gtm
hacktoberfest-2022
help
infra-dev-alerts
infra-dev-alerts-webhook
introductions
issue
kubernetes
license-questions
linen-test
local-brazil
local-london
local-sf
marcos-test-discourse
memes-and-banter
mos-client
mos-client
motleyfool
movedata-a-different-way-to-work
movedata-airbyte-connection-management-scale
movedata-airbyte-orchestration-in-gcp
movedata-better-data-testing-with-the-data-error-generating-process
movedata-bring-your-own-infra
movedata-building-a-real-time-user-facing-dashboard-with-airbyte-redpanda-and
movedata-building-connectors-in-minutes
movedata-cicd-for-data-building-devtest-data-environments-with-lakefs
movedata-data-analysts-are-setup-to-fail-and-its-our-fault
movedata-data-engineering-is-software-engineering-and-software-engineering-is-da
movedata-data-orchestration-is-not-just-running-jobs-on-a-schedule
movedata-dataops-on-the-open-modern-data-stack
movedata-dev-first-open-source-data-observability-solution
movedata-ducky-data-crunching-on-the-laptop---the-pendulum-swings
movedata-five-causes-of-data-quality-issues
movedata-future-evolution-of-the-data-stack
movedata-general
movedata-guardrails-not-stop-signs
movedata-if-you-build-it-will-they-come-how-to-activate-your-modern-data-stack
movedata-in-2023-data-trust-will-be-more-important-than-ever-before
movedata-ingesting-data-with-airbyte-into-a-high-speed-data-lakehouse-using-trin
movedata-keynote-building-the-foundations-of-data-movement
movedata-let-your-data-team-choose-their-own-tools
movedata-mobilize-the-worlds-data
movedata-modern-data-management---how-to-achieve-data-discovery
movedata-moving-data-reliably-ingestion-observability-are-better-together
movedata-navigating-the-modern-data-stack-with-open-source-headless-bi
movedata-open-source-communities-shape-modern-data-stacks
movedata-prefect-ing-self-hosted-airbyte
movedata-prep-your-pipelines---reverse-etl-and-the-coming-great-flood
movedata-speakers
movedata-streaming-made-modern
movedata-the-best-data-warehouse-is-a-lakehouse
movedata-the-data-ecosystem-is-ready-for-etl-to-be-dead
movedata-the-end-of-the-pipeline
movedata-traditional-data-catalogs-will-be-replace-by-active-metadata-platforms
movedata-using-airbyte-on-day-1of-our-startup
movedata-using-airbyte-to-build-your-dremio-open-data-lakehouse
movedata-who-needs-a-traditional-data-warehouse-when-you-can-get-real-time-analy
movedata-why-bi-is-not-enough
octavia-cli
octavia-life
office-hour
office-hour-12-october
oss-master-build-failure
p0-amazon-ads-03-30
p0-cloud-syncs-failing-dec-15
p0-instagram-incompatible-metrics
p0-source-airtable-stream-failures
partner-productpair-hightouch
prod1-alerts-test
prod1-health
random-news-and-events
shameless-plugs
troubleshooting
understanding-airbyte
using-the-cdk
write-for-the-community
Powered by LinenTitle
Question
How can I resolve the 'The table is full' error when trying to replicate data from MySQL to BigQuery?
Answer
One possibility is to clear the temporary tables and try to re-run the sync. Alternatively, you may need to increase the allocated memory for Aurora. Upgrading the connector to the latest version may also resolve the issue. For improving performance in loading big tables in MySQL, create a separate thread specifically for that issue on the same channel.
m
Maykon Lopes
10/05/2022, 12:12 AM[AWS Aurora MySQL /rdsdbdata/tmp is full]
Hey folks, I'm facing an issue to replicate data from a mysql to bigquery, the fail is really quickly in log output, showing this:
Caused by: java.lang.RuntimeException: java.sql.SQLException: The table '/rdsdbdata/tmp/#sql129d_35_1' is full
at io.airbyte.db.jdbc.StreamingJdbcDatabase.unsafeQuery(StreamingJdbcDatabase.java:79)
at io.airbyte.db.jdbc.JdbcDatabase.unsafeQuery(JdbcDatabase.java:195)
at io.airbyte.integrations.source.relationaldb.AbstractRelationalDbSource.lambda$queryTable$0(AbstractRelationalDbSource.java:61)
... 34 more
Caused by: java.sql.SQLException: The table '/rdsdbdata/tmp/#sql129d_35_1' is full
at com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:129)
at com.mysql.cj.jdbc.exceptions.SQLExceptionsMapping.translateException(SQLExceptionsMapping.java:122)I tried increase size in aurora rds mysql parameter group, but still having the issue. Any clue?
s
Saj Dider (Airbyte)
10/05/2022, 2:39 PMHey @Maykon Lopes, how much memory did you allocate for the aurora and what is the size of the sync in terms of records and bytes?
My guess is that you can clear the tmp tables and try to re-run the sync. Otherwise, you may not have increased the memory enough
m
Maykon Lopes
10/19/2022, 5:57 PMHey @Saj Dider (Airbyte) sorry my late answer here, the issue was solved after upgrade and connector to lastest version. Thanks for you help.
Now I'm struggling to improve performance in load big tables in mysql, tables with more than 300GB.
s
Saj Dider (Airbyte)
10/19/2022, 7:17 PMHey @Maykon Lopes, happy to hear that upgrading the connector fixed your issue. Create a separate thread for the performance issue with MySQL on this channel.