Airbyte is an open-source data integration engine that helps you consolidate your data in your data warehouses, lakes and databases.

Airbyte

Untitled.json

I'm attempting an initial sync on *Airbyte OS*, using the *Google Analytics* (Universal Analytics) connector version 0.1.30. The sync ran for about 12 hours and then failed with the error shown in the code snippet at bottom:

My authorization method = service account key, which in prior (shorter) attempts has never failed for me with `ACCESS_TOKEN_EXPIRED`. The 2nd sync attempt started right after the failure and is still going, so it's not clear that the token actually expired, or really that the token _ever_ expires. The service account JSON key is formatted like below, with sensitive values redacted.

I think my recourse is to chunk up the initial sync into pieces which take less time, then stitch together the results and reload the target table from there, and hope this doesn't confuse incremental sync.

Thoughts?

```{
  "type": "service_account",
  "project_id": "XXXXX",
  "private_key_id": "XXXXX",
  "private_key": "-----BEGIN PRIVATE KEY-----XXXXX-----END PRIVATE KEY-----\n",
  "client_email": "XXXXX",
  "client_id": "bigint",
  "auth_uri": "<https://accounts.google.com/o/oauth2/auth>",
  "token_uri": "<https://oauth2.googleapis.com/token>",
  "auth_provider_x509_cert_url": "<https://www.googleapis.com/oauth2/v1/certs>",
  "client_x509_cert_url": "<https://www.googleapis.com/robot/v1/metadata/XXXXX>"
}```

Hey maybe the access token requested internally has expired. Could you try with much closer start_date and see if that is successful?

Yes, I've succeeded when going back only as far as Jan 2021. Unfortunately I'm attempting to get back to Jan 2020. I wish there was a way to set an end date, in which case I could chunk up this problem (I think) into one year at a time.

So instead I'm planning to chunk up the problem by report, by setting up multiple connections instead of trying to sync 9 different streams from one connection.

Got it. Maybe you can use a script to get all the old data and load first and then later start using Airbyte so that it becomes easy.

Yes, in general I'm curious about the ability to load data from a source other than Airbyte, either before or after setting up an initial sync on Airbyte. It's not clear to me whether Airbyte incremental sync is likely to get confused by 'foreign' rows in a target table.

Hey got it. Yeah I understand that it is bit complex, you can also try to write a custom connector with end date if the Google API supports that. Incase you need some help there you can reach out to us

I guess I should ask more directly: is it possible to load a table either before or after initial sync with 'foreign' rows without causing the next incremental sync to throw an error?

I eventually got my GA sync to work going back all the way to Jan 2020. The two times it failed it ran for 10+ hours with midnight happening during that time. The more recent success ran for the same # of hours, but it all happened within one day. Maybe the token always expires when a long sync goes past midnight?

Awesome. Thanks a lot for the heads up :+1: