Hi, I am wondering if GA4 connector handles new da...
# replication-ask-ai
m
Hi, I am wondering if GA4 connector handles new data properly. I am running it everyday for custom reports, will it keep appending data for each pagepath even if data was already retrieved for that date and pagepath?
k
The Google Analytics 4 (GA4) connector in Airbyte supports several [sync modes](https://docs.airbyte.com/cloud/core-concepts#connection-sync-modes), including Full Refresh - Overwrite, Full Refresh - Append, Incremental - Append, and Incremental - Append + Deduped. If you are using the Incremental - Append or Incremental - Append + Deduped mode, new data will be appended to your existing data. However, please note that for custom reports, incremental syncs are supported only if you add the “ga:date” dimension to your custom report. Here is an example of how to add the "ga:date" dimension to a custom report:
Copy code
json [{"name": "new_users_per_day", "dimensions": ["ga:date","ga:country","ga:region"], "metrics": ["ga:newUsers"]}, {"name": "users_per_city", "dimensions": ["ga:city"], "metrics": ["ga:users"]}]
If you had already configured your connection without using Custom Reports in the source, it’s important to refresh the source schema in your replication settings, so the new streams are added. Please refer to the [Airbyte documentation](https://docs.airbyte.com/integrations/sources/google-analytics-data-api) and [tutorials](https://airbyte.com/tutorials/export-google-analytics-to-bigquery) for more details.
Based on the information provided in the extracts, Airbyte's incremental sync mode pulls only the data that has changed since the previous sync. This is usually implemented using a cursor value, such as an
updated_at
timestamp. Records whose
updated_at
value is less than or equal than that cursor value have been synced already, and the next sync should only export records whose
updated_at
value is greater than the cursor value. However, there are some limitations to this approach. If multiple modifications are made during the same day to the same records and the frequency of the sync is not granular enough (for example, set for every 24h), then intermediate modifications to the data are not going to be detected and emitted. Only the state of data at the time the sync runs will be reflected in the destination. In the case of Google Analytics, Airbyte makes data requests to Google in one day increments to minimize the chances of sampling being applied to your data. This reduces the amount of data the Google API processes per request, thus minimizing the chances of sampling being applied. However, this also increases the time it takes to export your Google Analytics data. So, if the
averagesessions
value for a
pagepath
was retrieved yesterday and has been modified slightly today, it will be updated correctly in the next incremental sync only if the sync is run after the modification has been made and the
updated_at
timestamp of the record is greater than the cursor value from the last sync. For more details, you can refer to the [Incremental Sync documentation](https://docs.airbyte.com/connector-development/connector-builder-ui/incremental-sync/) and the [Google Analytics Source documentation](https://docs.airbyte.com/integrations/sources/google-analytics-v4).
m
@kapa.ai Say i have a report with ga:date and averagesessions per pagepath. If the averagesessions was retrieved yesterday for that pagepath - if I run incremental refresh today and averagesession has been modified slightly (given google takes up to 48 hours to process data) - will that new averagesession be updated correctly to newest value?
k
A message from kapa.ai
Based on the information provided in the extracts, Airbyte's incremental sync mode pulls only the data that has changed since the previous sync. This is usually implemented using a cursor value, such as an
updated_at
timestamp. Records whose
updated_at
value is less than or equal than that cursor value have been synced already, and the next sync should only export records whose
updated_at
value is greater than the cursor value. However, there are some limitations to this approach. If multiple modifications are made during the same day to the same records and the frequency of the sync is not granular enough (for example, set for every 24h), then intermediate modifications to the data are not going to be detected and emitted. Only the state of data at the time the sync runs will be reflected in the destination. In the case of Google Analytics, Airbyte makes data requests to Google in one day increments to minimize the chances of sampling being applied to your data. This reduces the amount of data the Google API processes per request, thus minimizing the chances of sampling being applied. However, this also increases the time it takes to export your Google Analytics data. So, if the
averagesessions
value for a
pagepath
was retrieved yesterday and has been modified slightly today, it will be updated correctly in the next incremental sync only if the sync is run after the modification has been made and the
updated_at
timestamp of the record is greater than the cursor value from the last sync. For more details, you can refer to the [Incremental Sync documentation](https://docs.airbyte.com/connector-development/connector-builder-ui/incremental-sync/).
m
@kapa.ai Cool - in the airbyte UI I see cursor field for my custom reports is date, does that not mean that it would not retrieve any row when date is less than the latest fetched date? In which case we would miss a row from yesterday updated today no?
k
A message from kapa.ai
Yes, you're correct. If the cursor field is set to
date
, then Airbyte's incremental sync will only retrieve rows where the
date
is greater than the latest fetched date. This means that if a row from yesterday is updated today, it would not be retrieved in the incremental sync because its
date
is not greater than the latest fetched date. This is one of the limitations of using a cursor field for incremental sync. If modifications to the underlying records are made without properly updating the cursor field, then the updated records won't be picked up by the incremental sync as expected. The source connectors extract delta rows using a SQL query that looks like this:
Copy code
sql SELECT * FROM table WHERE cursor_field >= 'last_sync_max_cursor_field_value'
So, if the
cursor_field
is
date
and it's not updated when
averagesessions
is modified, then the updated
averagesessions
won't be picked up in the incremental sync. For more details, you can refer to the [Incremental Sync documentation](https://docs.airbyte.com/connector-development/connector-builder-ui/incremental-sync/) and the [Incremental Sync - Append documentation](https://docs.airbyte.com/understanding-airbyte/connections/incremental-append/).
m
@kapa.ai so custom reports in GA4 will never be fully accurate ?
k
A message from kapa.ai
The accuracy of custom reports in Google Analytics 4 (GA4) when using Airbyte largely depends on the configuration of the sync process and the nature of the data being synced. Airbyte's incremental sync mode is designed to pull only the data that has changed since the last sync. However, if the data changes after it has been synced and the cursor field (like
date
) does not reflect this change, then the updated data might not be picked up in the next sync. In the case of Google Analytics, data can take up to 48 hours to be processed and appear in your reports. If you run an incremental sync before the data is fully processed, the sync might not capture the most accurate data. Therefore, while Airbyte can help you sync your GA4 data, the accuracy of the data in your custom reports can be influenced by factors such as the timing of the sync and the processing time of Google Analytics data. For more accurate results, you might consider adjusting the sync frequency or the cursor field to better capture the changes in your data. For more information, you can refer to the [Google Analytics Source documentation](https://docs.airbyte.com/integrations/sources/google-analytics-v4) and the [Incremental Sync documentation](https://docs.airbyte.com/connector-development/connector-builder-ui/incremental-sync/).