I think I've opened an issue previous that was closed, but I'm still trying to understand behavior between Airbyte and Stitch for a Stripe source, more in 🧵

At some point in time, "updated" was added to the source schema, but not available as an incremental cursor

I opened an issue to see if we could move from created > updated

that issue was closed, updated is not in the Stripe API spec, no one knew how/why it got added

When using Stitch, it appears the singer tap is able to determine new vs updated records, and records an "updated" field with timestamp

my only option in Airbyte is to a do a full refresh, of every invoice/subscription/charge from the beginning of time to capture these updates

the incremental is only using created as a cursor, so for instance if a charge is created then later refunded, a full refresh is the only way to capture

however, Stitch/singer seems to be capturing these updates without issue

im curious if there's any way to get to a proper incremental load with the Stripe source

Original issue https://github.com/airbytehq/airbyte/issues/4921

Appears that Singer is using a 30 day lookback window, would that be something we could add?

rather than >= last cursor, >= last_cursor - 30 days

this way I can handle the late arriving updates in normalization. Bonus points if we can set the lookback window in the UI

omw to create a github issue

Thank you for listening to my TED talk 🙂

@Jeff Crooks sorry to ask you this but, could you transfer this to Github? At least we should you return about if possible/not implement the requested

@[DEPRECATED] Marcos Marx Already did! #5765 , @s responded saying this is not only feasible but possibly implemented quickly! Any news on the Mongo source update?

Also - I vaguely remember the ability to manually change the state? Is that a workaround I can use in the meantime?

yes, you could export the config and edit the job/attempts manually. I don't know if it's possible to do it now