https://linen.dev logo
Join Slack
Powered by
<@U04SE163WC8> We used the Stripe source to sync d...
# ask-ai
s
@kapa.ai We used the Stripe source to sync data from Stripe to Snowflake. However, after comparing the data in the Stripe portal and the data synced to Snowflake, we found many data discrepancies.
k
I'm sorry to hear that you're experiencing data discrepancies when syncing data from Stripe to Snowflake using Airbyte. There could be several reasons for this issue. 1. Incomplete Sync: As per the [tutorial on data replication](https://airbyte.com/tutorials/validate-data-replication-postgres-snowflake), replication jobs using incremental replication can sometimes halt midway, leading to data missing in the destination. 2. Changes Not Reflected: Changes in data at the source aren't reflected downstream, causing major discrepancies. 3. Data Structure: As per the [Github issue](https://github.com/airbytehq/airbyte/issues/30593), there could be a mismatch between the data structure in the source and the destination. For instance, the Stripe API does not expand subscriptions and sources by default, but this is reflected in the schema. To resolve these issues, you might want to: - Check your sync settings and ensure that you're using the correct replication method. If you're using incremental replication, you might want to switch to full refresh to capture any deletes or updates in your source data table. - Validate your data after replication to ensure that all data has been accurately replicated. You can use data-diff for this purpose as explained in the [tutorial](https://airbyte.com/tutorials/validate-data-replication-postgres-snowflake). - Check the schema of your data and ensure that it aligns with the data structure in the Stripe API. If the issue persists, I recommend raising a [Github issue](https://github.com/airbytehq/airbyte/issues) to get help from the Airbyte community.
s
@kapa.ai, can the source-defined cursor be updated? Because I found the bug is the wrong cursor field.
k
A message from kapa.ai
Yes, the source-defined cursor can be updated. However, it's important to note that if 
source_defined_cursor
in 
AirbyteStream
is set to true, then the source determines the cursor field internally and it cannot be overridden [source](https://docs.airbyte.com/understanding-airbyte/airbyte-protocol#logic-for-resolving-the-cursor-field). If you want to update the cursor field, you would need to set 
source_defined_cursor
to false and then set the 
cursor_field
in 
ConfiguredAirbyteStream
to the field you want to use as the cursor. If 
cursor_field
is not set, the 
default_cursor_field
in 
AirbyteStream
will be used. Here's the logic for resolving the cursor field: - If 
source_defined_cursor
in 
AirbyteStream
is true, then the source determines the cursor field internally. It cannot be overridden. If it is false, continue... - If 
cursor_field
in 
ConfiguredAirbyteStream
is set, then the source uses that field as the cursor. If it is not set, continue... - If 
default_cursor_field
in 
AirbyteStream
is set, then the sources use that field as the cursor. If it is not set, continue... - Illegal - If 
source_defined_cursor
, 
cursor_field
, and 
default_cursor_field
are all false-y, this is an invalid configuration. Please note that the cursor field should be of the correct type as per the [source-defined cursor documentation](https://docs.airbyte.com/understanding-airbyte/connections/incremental-append#source-defined-cursor). Also, nested cursor fields might not be supported due to restrictions for supporting normalization [source](https://github.com/airbytehq/airbyte/issues/36253).
s
@kapa.ai What if I find the source_defined_cursor is wrong? Where can i report this issue?
k
A message from kapa.ai
If you find that the 
source_defined_cursor
is incorrect, you can report this issue on the Airbyte Github repository. Here are the steps to do so: 1. Visit the [Airbyte Github Issues page](https://github.com/airbytehq/airbyte/issues). 2. Click on the "New issue" button. 3. Choose the appropriate issue template. If you're not sure which one to choose, you can use the "Bug report" template. 4. Fill in the details of the issue in the template. Be sure to include as much information as possible, such as the source connector you're using, the version of Airbyte you're running, and detailed steps to reproduce the issue. Also, include the details about the 
source_defined_cursor
being incorrect. 5. Click "Submit new issue" to create the issue. The Airbyte team and community will then be able to see your issue and provide help.
Open in Slack
PreviousNext
Powered by