https://linen.dev logo
k

Kyle Mok

02/07/2022, 4:49 PM
it would also be helpful to have compound cursors for incremental loading (select multiple fields)
You can have multiple cursor_fields My bad but why do you need this? Is this a requirement for custom connector?
@Harshith (Airbyte) This is interesting. how can we do it with mysql connector?
@Raj I think my bad I got confused between primary keys and cursors 🤣 (just today thing). Sorry to get you excited
But now I am excited how will this feature help?
@Harshith (Airbyte) In many systems timestamps are not maintained properly sometimes we might need a custom logic like createdAt and updateAt as the cursor. This is just an example. There can be other cases aswell
Got it. Then it would makes sense if we can create an issue and so that team can track this
r

Rachel RIZK

02/08/2022, 3:37 PM
my specific use case is that we have a table with two date fields,
created_at
and
discarded_at
. setting the cursor on
created_at
will capture records created but if it ends up becoming
discarded
, we won't know. On the other hand if we track
discarded_at
, we won't see when the record is
created
. An option on our engineering side is to have an
updated_at
field, but that may or may not be prioritized, so if we could
coalesce(discarded_at, created_at)
as a cursor, that could help solve the problem
g

gunu

02/09/2022, 4:03 AM
Got it. Right now the option is to handle this in Custom DBT but it would also makes sense if you can create a issue around it add this info
t

Tyler Buth

02/10/2022, 4:28 PM
Is there Custom DBT executed prior to loading?
d

Dave Lindley

02/11/2022, 7:03 PM
No, custom dbt are executed after the loading process
a

Arvi

02/14/2022, 4:29 PM
got it, that means that custom dbt won't help here since the data won't be loaded
You’re correct
2 Views