https://linen.dev logo
Join Slack
Powered by
it would also be helpful to have compound cursors ...
# feedback-and-requests
k
it would also be helpful to have compound cursors for incremental loading (select multiple fields)
You can have multiple cursor_fields My bad but why do you need this? Is this a requirement for custom connector?
@Harshith (Airbyte) This is interesting. how can we do it with mysql connector?
@Raj I think my bad I got confused between primary keys and cursors 🤣 (just today thing). Sorry to get you excited
But now I am excited how will this feature help?
@Harshith (Airbyte) In many systems timestamps are not maintained properly sometimes we might need a custom logic like createdAt and updateAt as the cursor. This is just an example. There can be other cases aswell
Got it. Then it would makes sense if we can create an issue and so that team can track this
r
my specific use case is that we have a table with two date fields, 
created_at
and 
discarded_at
. setting the cursor on 
created_at
will capture records created but if it ends up becoming 
discarded
, we won't know. On the other hand if we track 
discarded_at
, we won't see when the record is 
created
. An option on our engineering side is to have an 
updated_at
field, but that may or may not be prioritized, so if we could 
coalesce(discarded_at, created_at)
as a cursor, that could help solve the problem
g
Got it. Right now the option is to handle this in Custom DBT but it would also makes sense if you can create a issue around it add this info
t
Is there Custom DBT executed prior to loading?
d
No, custom dbt are executed after the loading process
a
got it, that means that custom dbt won't help here since the data won't be loaded
You’re correct
2 Views
Open in Slack
PreviousNext
Powered by