https://linen.dev logo
Join Slack
Powered by
Question: is there a provision to select multiple ...
# replication-troubleshooting
p
Question: is there a provision to select multiple columns as cursor field in case of incremental updates to postgres
a
Not currently-- can you share a bit more about how you’d expect multiple cursors to work?
p
Hi Andy,
Thank you for responding back, we have data in tables with unique key as multiple columns like classification_code+ type
as unique row
then what is way to handle incremental update, is it something like dbt need to be involved
a
Oh got it! I think what you might be looking for is how to set multiple 
primary keys
, which we do support. The 
cursor
helps us identify what rows have been added or updated since the last sync. A common 
cursor
column is something like an 
updated_at
timestamp column.
p
yes, i am looking for multiple primary keys, there is no date field to identify which got updated/created/deleted
data should be compared using composite key and then add/delete
could you help how multple primary keys can be set
ya, i am able to select multiple columns in primary key
but it helps only for deduplication right, actual incremental logic works based on cursor field
v
I think what you’re looking for is CDC or 
full_fresh
. Cause if you only rely on composite key to identify new/updated/deleted rows, you must always read the whole table & compare with destination. Not different compared with full_refresh.
p
Hi Vu, thank you, i got your point, thank you, i will check CDC replication. full_refresh may not be feasible as data is growing.
v
I’m glad it helps
a
Yup! CDC is the next best option. Thanks for jumping in, Vu!
p
Hi Andy, i am facing another critical issue to proceed with airbyte implementation, we are checking data to move from CSV to postgres
unfortunately all numbers are automically storing as float
any pointers on this
unless this is solved i cannot proceed
5 Views
Open in Slack
PreviousNext
Powered by