Hi All, I am creating a connector for a system tha...
# contributing-to-airbytew
Wisse Jelgersma
12/28/2021, 12:22 PMHi All, I am creating a connector for a system that can also list the deleted rows from a certain point in time. Is there a way to mark rows as deleted in an incremental stream? Would it work if I add ab_cdc_deleted_at and ab_cdc_updated_at columns to my output?
Wisse Jelgersma
12/28/2021, 1:48 PMHi @Wisse Jelgersma, do you want deleted rows to not be part of the normalized output or do you simply want to flag them as deleted?
Wisse Jelgersma
12/28/2021, 1:57 PMWe would prefer it be removed in the final table. This is what happens with CDC sources as well, right?
Wisse Jelgersma
12/28/2021, 3:13 PMYes, I think so. @Chris (deprecated profile) I believe this is an exciting normalization question: will basic normalization for CDC work if the source connector used is not a JDBC-based source? Is a source considered as CDC only according to its schema?
Wisse Jelgersma
12/28/2021, 5:32 PMHi,
I have a same question that I want to remove rows that are 30 days old. What would be preferable way, when using airbyte with postgres?
u
[DEPRECATED] Marcos Marx
12/28/2021, 10:31 PM
@Zawar khan will transfer your data from your source to destination, so one way is to use a view that only has 30 days and use full refresh (in case the table is not large) other way is to use dbt to post-transform your data.
b
Boggdan Barrientos
12/29/2021, 8:27 AM@[DEPRECATED] Augustin Lafanechere I see @Chris (deprecated profile) is on holiday, is there anyone else in the Airbyte team that can answer my question in the meantime? I think you formulated my question perfectly.
c
Chris (deprecated profile)
01/03/2022, 5:51 PM
Would it work if I add ab_cdc_deleted_at and ab_cdc_updated_at columns to my output?Yes, that should probably work
m
Mukul Gopinath
01/04/2022, 10:14 AMThanks @Chris (deprecated profile), I will try it out
s
Serhii Chvaliuk [GL]
01/06/2022, 4:31 PMSo far it looks like it's working! thanks @Chris (deprecated profile) and @[DEPRECATED] Augustin Lafanechere
Serhii Chvaliuk [GL]
01/06/2022, 4:31 PMThank you @Wisse Jelgersma for the experimentation 😄
9 Views