Hello, I'm trying to sync a large table of about 6...
# ask-community-for-troubleshootingp
Paul Rus
11/07/2022, 9:00 PMHello, I'm trying to sync a large table of about 6 GB from BigQuery to Postgres, however I've run into an error right near the end of the sync
Paul Rus
11/07/2022, 9:01 PMThis is the error:
compiled SQL at ../build/run/airbyte_utils/models/generated/airbyte_incremental/scd/for_export/contacts_common_part_10000000_scd.sql
2 of 3 ERROR creating incremental model for_export.contacts_common_part_10000000_scd.................................... [ERROR in 7566.15s]
Database Error in model contacts_common_part_10000000_scd (models/generated/airbyte_incremental/scd/for_export/contacts_common_part_10000000_scd.sql)
temporary file size exceeds temp_file_limit (3807390kB)
compiled SQL at ../build/run/airbyte_utils/models/generated/airbyte_incremental/scd/for_export/contacts_common_part_10000000_scd.sql,retryable=<null>,timestamp=1667853360184], io.airbyte.config.FailureReason@7b55f0a6[failureOrigin=normalization,failureType=system_error,internalMessage=temporary file size exceeds temp_file_limit (3807390kB),externalMessage=Normalization failed during the dbt run. This may indicate a problem with the data itself.,metadata=io.airbyte.config.Metadata@3fc7abf6[additionalProperties={attemptNumber=0, jobId=804129, from_trace_message=true}],stacktrace=AirbyteDbtError:
2 of 3 ERROR creating incremental model for_export.contacts_common_part_10000000_scd.................................... [ERROR in 7566.15s]
Database Error in model contacts_common_part_10000000_scd (models/generated/airbyte_incremental/scd/for_export/contacts_common_part_10000000_scd.sql)
temporary file size exceeds temp_file_limit (3807390kB)
compiled SQL at ../build/run/airbyte_utils/models/generated/airbyte_incremental/scd/for_export/contacts_common_part_10000000_scd.sql
2 of 3 ERROR creating incremental model for_export.contacts_common_part_10000000_scd.................................... [ERROR in 7566.15s]
Database Error in model contacts_common_part_10000000_scd (models/generated/airbyte_incremental/scd/for_export/contacts_common_part_10000000_scd.sql)
temporary file size exceeds temp_file_limit (3807390kB)
compiled SQL at ../build/run/airbyte_utils/models/generated/airbyte_incremental/scd/for_export/contacts_common_part_10000000_scd.sql,retryable=<null>,timestamp=1667853360184]]]
2022-11-07 203600 normalization-orchestrator >
2022-11-07 203600 normalization-orchestrator > ----- END DEFAULT NORMALIZATION -----
2022-11-07 203600 normalization-orchestrator >
2022-11-07 203600 normalization-orchestrator > Writing async status SUCCEEDED for KubePodInfo[namespace=jobs, name=orchestrator-norm-job-804129-attempt-0, mainContainerInfo=KubeContainerInfo[image=airbyte/container-orchestrator:dev-4c6f520, pullPolicy=IfNotPresent]]...
2022-11-07 203601 INFO i.a.c.t.TemporalUtils(withBackgroundHeartbeat):316 - Stopping temporal heartbeating...
2022-11-07 203601 INFO i.a.w.t.TemporalAttemptExecution(get):162 - Stopping cancellation check scheduling...
2022-11-07 203601 INFO i.a.c.t.TemporalUtils(withBackgroundHeartbeat):283 - Stopping temporal heartbeating...
Paul Rus
11/07/2022, 9:02 PMIt seems to me like the temp files size limit has been reached during the sync
Paul Rus
11/07/2022, 9:03 PMWhat do you recommend I could try, since this table is planned to be continuously expanded in BigQuery, requiring constantly bigger syncs to Postgres?
s
Sunny Hashmi (Airbyte)
11/08/2022, 4:07 PMHey there, can you attach the full sync log in case there is more information to help us understand what happened?
From what I can tell so far this error looks to be from the destination postgres. Depending on how you have it configured, you may need to add more disk space or adjust the `temp_file_limit`: https://www.postgresql.org/docs/current/runtime-config-resource.html#RUNTIME-CONFIG-RESOURCE-DISK
p
Paul Rus
11/09/2022, 1:01 PMThis is the log from Airbyte
u
user
12/06/2022, 8:24 PMHello Paul Rus, it's been a while without an update from us. Are you still having problems or did you find a solution?