Hello. We are using the mixpanel integration to pull data down. The problem we are having is that it seems to always pull down all records, even when it is set on incremental sync. Each day it pulls 1.8m records. Is there anything we can change about the integration to make it only get the newest data?

Yes, there are a few things you can do to ensure that your Mixpanel integration only pulls the newest data. Firstly, you need to ensure that your Mixpanel source connector is set to the correct sync mode. According to the [Mixpanel source connector documentation](https://docs.airbyte.com/integrations/sources/mixpanel), it supports the following sync modes: - Full Refresh - Overwrite - Full Refresh - Append - Incremental - Append - Incremental - Append + Deduped For your use case, you should use either the Incremental - Append or Incremental - Append + Deduped sync mode. These modes pull only the data that has changed since the previous sync, as opposed to all the data available in the data source. However, please note that the Incremental sync returns duplicated (old records) for the state date due to API filter limitation, which is granular to the whole day only. This might be the reason why you are seeing all records being pulled down. To implement incremental sync, you need to use a cursor value (like a timestamp) that delineates which data was pulled and which data is new. A common cursor value is an

updated_at

timestamp. This cursor means that records whose

updated_at

value is less than or equal than that cursor value have been synced already, and that the next sync should only export records whose

updated_at

value is greater than the cursor value. Here is an example of how to set the sync mode to incremental in your `configured_catalog.json`:

json "sync_mode": "incremental",

After setting the sync mode, you can try running your connector again:

bash python main.py read --config secrets/config.json --catalog sample_files/configured_catalog.json

To verify that incremental sync is working, pass the input state back to the connector and run it again:

bash # Save the latest state to sample_files/state.json   python main.py read --config secrets/config.json --catalog sample_files/configured_catalog.json | grep STATE | tail -n 1 | jq .state.data > sample_files/state.json      # Run a read operation with the latest state message   python main.py read --config secrets/config.json --catalog sample_files/configured_catalog.json --state sample_files/state.json

You should see that only the record from the last date is being synced. This is acceptable behavior, since Airbyte requires at-least-once delivery of records, so repeating the last record twice is OK. If you still encounter issues, you may want to check out the [incremental sync documentation](https://docs.airbyte.com/understanding-airbyte/connections/incremental-append) for more information on different types of configuration.