Hey guys, I’ve been using airbyte to sync up my da...
# ask-community-for-troubleshootingr
Rahul Padhy
05/27/2021, 8:12 AMHey guys, I’ve been using airbyte to sync up my data from S3 source to dump into BigQuery. Now, I’ve made 6 different connectors that sync up to give data in 6 different tables in BigQuery in the same dataset. But 1 of the connectors is failing to sync up data for about the past couple of days. Interestingly, this connector was working fine before that. I skimmed through the logs and from there, I got to know that S3 endpoint URL for that source. But if that’s that the case, I’m unable to understand as to how things were working fine before that and if we keep the URI aside, the other 5 connectors have the same settings and they are working fine. I’m attaching the related snapshots and the log file below in this conversation thread.
✅ 1
Rahul Padhy
05/27/2021, 8:12 AMlogs-112-0.txt
Rahul Padhy
05/27/2021, 8:13 AMScreenshot 2021-05-27 at 1.43.01 PM.png
Rahul Padhy
05/27/2021, 8:17 AMUPDATE : Just noticed that there were data sync-up failures with other connectors as well
Rahul Padhy
05/27/2021, 8:17 AMPutting up the snapshots below…
Rahul Padhy
05/27/2021, 8:17 AMScreenshot 2021-05-27 at 1.46.05 PM.png,Screenshot 2021-05-27 at 1.46.22 PM.png,Screenshot 2021-05-27 at 1.46.50 PM.png
Rahul Padhy
05/27/2021, 9:34 AMCan anyone please suggest what might be going wrong?
d
Davin Chia (Airbyte)
05/27/2021, 10:15 AM
does this happen regularly?
Davin Chia (Airbyte)
05/27/2021, 10:15 AM
it might just be a transient networking blip somewhere
r
Rahul Padhy
05/27/2021, 10:16 AMNo actually, as can be seen from above snapshots, there have been a couple of days wherein this issue was observed. Otherwise, before and after that, things were getting synced up fine.
Rahul Padhy
05/27/2021, 10:17 AMBut for 1 particular source, issues have been arising for the past 3 days, so decided to ask here..
d
Davin Chia (Airbyte)
05/27/2021, 10:17 AM
Gotcha
Davin Chia (Airbyte)
05/27/2021, 10:17 AM
Can you think of a pattern of how the syncs are run?
r
Rahul Padhy
05/27/2021, 10:18 AMPattern as in? They have been set to do a full overwrite once every 24 hours
d
Davin Chia (Airbyte)
05/27/2021, 10:18 AM
And they are all on the same machine?
r
Rahul Padhy
05/27/2021, 10:18 AMYes
d
Davin Chia (Airbyte)
05/27/2021, 10:18 AM
Maybe the syncs run at the same time and they are using up some networking resources?
r
Rahul Padhy
05/27/2021, 10:24 AMOh okay, didn’t know if that’s possible, not sure about that actually.
u
[DEPRECATED] Marcos Marx
05/27/2021, 10:31 PM
@Rahul Padhy those errors keep happening? (just checking) this files are multi-region? are frequently updated?
r
Rahul Padhy
05/28/2021, 2:44 PMFor 1 of the connectors, it happened and further syncing hasn’t been done so far
Rahul Padhy
05/28/2021, 2:44 PMAll the files are in the same region in AWS S3
Rahul Padhy
05/28/2021, 2:47 PMApart from that 1 connector, others have been syncing up nicely thus far - basically the issue is that they are unable to access the S3 endpoint (URL)
u
[DEPRECATED] Marcos Marx
05/28/2021, 11:01 PM
thats something interesting... is that file structure (size, columns) different from the others? just random questions, hope the next sync it work
r
Rahul Padhy
05/29/2021, 11:41 AMYes, the file structure differs across different sources w.r.t. number of rows, number of columns.
4 Views