Hello, I am running Airbyte Open Source on GCP (Go...
# replication-troubleshooting
c
Hello, I am running Airbyte Open Source on GCP (Google Cloud Engine) where source is Bing Ads and destination is BIgQuery. For GCE VM instance I am using n1-standard-2 with boot disk debian-10-buster-v20221206 (size 10 GB). I also added additional disk with 40GB size because I thought maybe I needed more space. (which don’t seem to be in use). When I try to sync data from Bing Ads worth 3 month data, it crash with error message: “Failure Origin: destination, Message: Something went wrong in the connector. See the logs for more detail.“. What could be the issue?
👀 1
m
The Bing Ads source raised this error:
suds.WebFault: Server raised fault: 'Invalid client data. Check the SOAP fault details for more information. TrackingId: 19a2e505-6fc7-4bc2-8afe-e4352e8a37d8.
which is not very informative 😞
c
Does it mean the problem is happening on Bing Ads? Also, why is “JSON schema validation failed” happening?
m
It looks like the data returned by Bing is corrupted somehow
c
I started having this issue only recently. I realize that there was an update to Bing Ads connector version on January 30, 2023. Would it be possible there was a bug introduced during update? https://docs.airbyte.com/integrations/sources/bing-ads/
I looked at my successful syncs I still see “JSON schema validation failed” message. So I think the error is something else
I manually tried to run it, and the sync succeeded this time. I changed nothing, I really don’t know what is happening any more
u
It is really strange, let me know if the error returns Chris!
c
Ran into the same error on the initial run, and then manually ran it and it worked. The following runs still take a while for the
ads
stream. Any suggestions for improving the performance?
u
Carolina, do you have a separate message explaining your situation?
c
I initially had this error, that you posted above. It went away after I stopped the initial sync, then ran the sync without it, and then re-added the ads stream back in. Now I have the stream running, but it takes anywhere from 45 mins to an hour with the majority of the time spent on
ads
. This is our first integration with Bing Ads, so I’m not super familiar with the data, but it looks like there is no option to run ads as an incremental with the current connector. Just wondering if this is expected or if there are suggestions to improve the performance.
Copy code
suds.WebFault: Server raised fault: 'Invalid client data. Check the SOAP fault details for more information. TrackingId: 19a2e505-6fc7-4bc2-8afe-e4352e8a37d8.
c
I am having another issue which may be unrelated to the original problem I was having: airbyte/db keeps crashing. I looked at the Disk Space Utilization, and it has reached 100%. Running Airbyte seem to create some stuff that use up the disk space, and for some reason does not get deleted at the end of the sync. As I run it multiple times the available space becomes smaller and smaller and at some point, it crashes. UPDATE: it looks like workspace volume is taking up all the space
u
What version of Airbyte are you running Chris? Carolina, I didn't find any issue in Github about Bing Ads performance. You can create a new one there. https://github.com/airbytehq/airbyte/issues?q=is%3Aissue+is%3Aopen+bing+ads For the SOAP I strongly recommend to send a message to Bing Ads support team to investigate what is the problem.
c
@Marcos Marx (Airbyte) I am using 0.40.32. For Airbyte using up all my disk spaces, I may have found a solution in another thread, and the solution was to use
TEMPORAL_HISTORY_RETENTION_IN_DAYS
(https://github.com/airbytehq/airbyte/pull/16247) which I am testing now. However, the issue where the sync randomly just fails is still unsolved. (It works sometimes and sometimes it doesn’t :S)
u
Chris, the problem with the sync is only happening with Bing Ads correct? You have others one, are they working?