@[DEPRECATED] Marcos Marx - Looks like the Stripe schema has been updated to add more fields and tables... Of note, the "updated" column has been added, but I can't select for cursor field. Is there any way around this? Its coming across as string instead of int

and appears the connection is setting all of the cursor fields and i cant change them

when cursor fields are defined by the source, airbyte UI doesn’t allow you to edit them no

it’d be interesting to discuss why you would need the ability to edit the cursor field for these use cases though! probably worth going over it in a github issue

so biggest thing is subscriptions create date wont ever change

but updated will

so if thy go from active > expired, I'd like to use "updated" as the cursor vs "created"

its strange because 1 - several of the tables weren't there before, and the updated column wasnt included

2 - other tools (Stitch, fivetran) seem to leverage this updated field by default, rather than created

when you say "cursor fields are defined by the source" is that a stripe definition or an airbyte source definition?

I meant it’s the airbyte source connector implementation for a source

Are you reporting a misuse of columns as a cursor on the stripe source? maybe it’s similar to this issue and we also need to correct it on another source too? https://github.com/airbytehq/airbyte/issues/4505

yep that sounds right

once the connector is implemented with

updated_at

columns as the source defined column for incremental, the user woudn’t need to modify it (and set it to

created_at

instead for example)?

I'll log a new issue for the stripe tables later today - curious, whats the reasoning behind preventing a user from changing it?

and yes thats correct

on initial creation created = updated

I’ll log a new issue for the stripe tables later today

Thanks! that will make sure we can fix that wrong choice for the many users of that source too!

curious, whats the reasoning behind preventing a user from changing it?

@s do you have a better answer maybe?

agree starting with best practice or tested config is best, but being flexible would be awesome. Maybe some sort of warning that "Hey We Dont Recommend This" when you change, but then still allowing the change

promise I'll have a Stripe Recipe + dbt Normalization to share once this is in 🙂

 curious, whats the reasoning behind preventing a user from changing it?

it’s a nontrivial footgun. First, not all fields are valid cursors. You can usually filter by created_at or sorted_at in an API, but not ID for example. There are also some invariants a cursor field must have that may not hold for other fields e.g: it must be comparable, it can’t be null, etc.. so allowing a user to configure it had some non trivial risk. Also, not all fields can be used as cursors. from a development standpoint, sources which can assume a certain cursor field will always be used are a little simpler than those which have to check which one was used, validate it, and use it etc. Neither of these is insurmountable, but given those issues, it wasn’t a big priority to do this. i like your idea of warning but letting user change anyways Would you be open to create a ticket expressing this?

yep I'll include a separate ticket aside form the stripe connector. Blocked some time this afternoon 🙂 Agreed it is a footgun if you don't understand the implications. Thanks!