https://linen.dev logo
Join Slack
Powered by
<@U04SE163WC8> According to <this> doc, the last ...
# replication-ask-ai
a
@kapa.ai According to this doc, the last entry is always re-normalized in incremental, even though it has not been modified. But when I practice it in Airbyte (hosted within a docker on an EC2), the last entry is not re-normalized in the destination after I ran a sync without any changes in source. (Postgres Source and Postgres destination). I can see that the last record's 
_airbyte_normalized_at
hasnt changed, anything I am missing?
k
Based on the information provided in the extracts, the 
_airbyte_normalized_at
field should indeed reflect the last time a record has been transformed and written by normalization, even if no modifications were made to the source data. This is because normalization of the most recent records in the destination is performed on each sync, as mentioned in the [tutorial on incremental data synchronization](https://airbyte.com/tutorials/incremental-data-synchronization). However, there are a few things that could potentially affect this: 1. The sync mode: The behavior you described is expected when the source is configured with sync modes compatible with incremental transformations such as full_refresh_append, incremental append, or incremental deduped history. If you're using a different sync mode, this might explain the discrepancy. 2. Normalization settings: If you have [Basic Normalization](https://docs.airbyte.com/understanding-airbyte/basic-normalization/) enabled, Airbyte will create a normalized table by extracting data from the JSON object. If normalization is not enabled, the 
_airbyte_normalized_at
field might not be updated. 3. Data transfer: Only rows that have changed in the source are transferred over the network and written by the destination connector. If no changes were made to the source data, the 
_airbyte_normalized_at
field might not be updated. If you've checked these factors and are still experiencing the issue, it might be worth looking into the [troubleshooting guide](https://airbyte.com/tutorials/creating-duckdb-destination-with-python) or reaching out to the Airbyte community for further assistance.
Based on the information provided in the [extracts](https://airbyte.com/tutorials/incremental-data-synchronization), the 
_airbyte_normalized_at
value should indeed update for the most recent records in the destination on each sync, even if there are no modifications in the source data. This is because normalization of the most recent records in the destination is performed on each sync. However, if you're not seeing this behavior, there could be a few reasons: 1. There might be an issue with the sync operation itself. Ensure that the sync operation is completing successfully without any errors. 2. The behavior might be affected by the specific sync mode you're using. For instance, in the [Incremental Sync - Deduped History](https://docs.airbyte.com/understanding-airbyte/connections/incremental-deduped-history) mode, a primary key is used to select only the most recent version of a given record for inclusion in the final deduplicated table. If you're using this mode, ensure that the primary key is correctly defined. 3. There might be an issue with the normalization process. Normalization in Airbyte can be run incrementally or in full refresh. The 
_airbyte_normalized_at
value serves to track when a record has been transformed and written by normalization. If this value isn't updating, it might indicate an issue with the normalization process. If you've checked these possibilities and are still experiencing the issue, I would recommend reaching out to the Airbyte community on their [Slack channel](https://airbytehq.slack.com) or [forum](https://discuss.airbyte.io) for further assistance.
a
@kapa.ai I want to able to select 
updated_at
as the cursor field. But I am unable to click on the cursor field radio button present next to 
updated_at
. How do I select it ?
k
Sorry @Abhra Gupta, it looks like your question contains files, which I cannot process. Please reformulate your question without including any files.
a
@kapa.ai I want to able to select a column called 
updated_at
as the cursor field. But I am unable to click on the cursor field radio button present next to 
updated_at
. How do I select it ?
k
A message from kapa.ai
Based on the information provided in the extracts, to select a column as the cursor field, you need to do it in the sync settings dropdown. This is particularly relevant for sources like Postgres where the user needs to choose which column in a database table they want to use as the 
cursor field
[source](https://docs.airbyte.com/understanding-airbyte/connections/incremental-append/). If you are unable to select the 
updated_at
column as the cursor field, it could be due to a few reasons: 1. The 
updated_at
column might not be in a compatible data type. The cursor field typically requires a data type that can be compared to track new or updated records, such as a timestamp or a number. 2. There might be an issue with the UI. Try refreshing the page or restarting your browser. 3. There might be an issue with your Airbyte instance. Try restarting your Airbyte instance. If you're still having trouble, you might want to check out the [Airbyte documentation](https://docs.airbyte.com/understanding-airbyte/connections/incremental-append/) or reach out to the Airbyte community for further assistance.
2 Views
Open in Slack
PreviousNext
Powered by