Hi, I am trying to sync data from Bing Ads to BigQ...
# replication-troubleshooting
c
Hi, I am trying to sync data from Bing Ads to BigQuery. I created VM on GCE and sync failed. I increased the disk size and it seems to work, and it failed the next day. I realized that the disk space is constantly on rise. It seems that Airbyte sync uses up some disk space in the process, and the disk space not deleted after finished(?). The question is: does the disk space used in the previous sync get erased before new one runs the next time? Or do you have to somehow manually delete it, or set it to be deleted in the config file somewhere?
s
Hi, I work with Postgres destination and if an Airbyte failure occurs, tmp tables stay and I have to drop them "manually", airbyte does not handle that. Maybe that's similar in your case. https://discuss.airbyte.io/t/is-it-okay-to-delete-the-airbyte-tmp-tables/1388
c
Yeah I had to delete tmp file from BigQuery too. But here I was talking about the actual disk space for VM on GCE
n
Hi, Chris! I believe there are some sync logs/other files that can be deleted after syncs, but are not deleted automatically. Let me dig around for the correct issue, I know I've seen it...
c
Any updates? Mine finally reached 100% and now it crashed…
n
@Chris, which version of Airbyte are you on?
c
@Nataly Merezhuk (Airbyte) I am using 0.40.32. I had to delete the workspace volume which erased the logs to get back the space..
n
Gotcha! I looked around and found this: https://github.com/airbytehq/airbyte/pull/16247 You should be able to use this env variable to delete old log files:
TEMPORAL_HISTORY_RETENTION_IN_DAYS
đź‘€ 1
c
Thanks! I will trying using that