https://linen.dev logo
Join Slack
Powered by
a
message has been deleted
c
@Chris (deprecated profile) my understanding is that this invalid. that if syncmode is full_refresh then destination syncmode should be overwite. if i'm wrong then we need to add more instructions in that github issue.
because in the github issue it only shows 3 states.
c
from the github issue, I am saying that there are niche use cases where this is needed: https://github.com/airbytehq/airbyte/issues/2371#issuecomment-795237466 So it is not really invalid and if we could support it as a fourth option, that would be great for some users
in the github issue, I did write: The supported combinations of source x destination sync modes are: • Full Refresh x overwrite: same as before • Full Refresh x append: source is transmitting the full content of the stream and destination keep appending to the same table (useful to detect deletions from one sync to the next, until CDC is supported with incremental) • Incremental x append: same as before • Incremental x append_dedup: the final table is deduplicated by the primary key but a new table with suffix 
_scd
is also created and retain the full history (same as Incremental - Append)
a
Well if we don’t support it - it means user won’t be able to use such destinations as he won’t be able to select value in drop down. We should support all cases on FE which we receive from backend. Otherwise it doesn’t make sense to release it
c
yes let’s support that case on FE please!
c
What should it be called?
As far as I can tell it is not in the mock up.
c
What should it be called?
Does 
Full Refresh - append
sounds fine?
a
so we won’t have any pk or cursor here for such case?
c
No, that would be needed if we were to do 
Full Refresh - aeduped + story
but we are not exposing that option (yet)
c
that sounds fine with me. i don't really have an opinion on this. just want to make sure we are giving artem the info he needs.
maybe it would be good to add to that spreadsheet this new option
as long as add columns describing any other expectations for these configurations.
e.g. expects primary key or not.
chris made you an editor on it in case it's helpful to start from there.
c
I changed everything 🙄
🤣 1
c
dope. seems clear to me now!
artem, is that clear enough to unblock you?
a
yep. Thank you all. Looks like now I have enough info
👍 1
Just 1 more thing to clarify now about cursors and pk: so if source has a sourceDefinedCursor then we should display disabled radio buttons for cursor, right? + in case of 
defaultCursorField
we want to have those field selected by default
c
1. For cursors that makes sense yes 2. For PK, i don’t know if we need to disable the options in case 
sourceDefinedPrimaryKey
is provided by the source… maybe a user could want to dedup using a different key and override the setting? so i would leave it editable but with the fields selected by default
👍 1
c
sourceDefinedPrimaryKey=true
then cursor fields should not be editable.
that selection overrides the users ability to select a cursor field.
this is how the system currently works. we can debate whether we want to keep it that way, but it's out of scope for this conversation. this is how it currently works.
ugh. sorry i'm getting confused between this conversation and the one in the main channel.
will cross out the things i said here taht don't make sense.
2 Views
Open in Slack
PreviousNext
Powered by