https://linen.dev logo
#feedback-and-requests
Title
# feedback-and-requests
j

Jeff Crooks

08/31/2021, 6:22 PM
I think I've opened an issue previous that was closed, but I'm still trying to understand behavior between Airbyte and Stitch for a Stripe source, more in ๐Ÿงต
u

user

08/31/2021, 6:22 PM
At some point in time, "updated" was added to the source schema, but not available as an incremental cursor
u

user

08/31/2021, 6:23 PM
I opened an issue to see if we could move from created > updated
u

user

08/31/2021, 6:23 PM
that issue was closed, updated is not in the Stripe API spec, no one knew how/why it got added
u

user

08/31/2021, 6:23 PM
When using Stitch, it appears the singer tap is able to determine new vs updated records, and records an "updated" field with timestamp
j

Jeff Crooks

08/31/2021, 6:24 PM
my only option in Airbyte is to a do a full refresh, of every invoice/subscription/charge from the beginning of time to capture these updates
u

user

08/31/2021, 6:24 PM
the incremental is only using created as a cursor, so for instance if a charge is created then later refunded, a full refresh is the only way to capture
u

user

08/31/2021, 6:25 PM
however, Stitch/singer seems to be capturing these updates without issue
u

user

08/31/2021, 6:25 PM
im curious if there's any way to get to a proper incremental load with the Stripe source
u

user

08/31/2021, 6:49 PM
u

user

08/31/2021, 6:50 PM
Appears that Singer is using a 30 day lookback window, would that be something we could add?
j

Jeff Crooks

08/31/2021, 6:51 PM
rather than >= last cursor, >= last_cursor - 30 days
u

user

08/31/2021, 6:55 PM
this way I can handle the late arriving updates in normalization. Bonus points if we can set the lookback window in the UI
u

user

08/31/2021, 6:56 PM
omw to create a github issue
u

user

08/31/2021, 7:40 PM
Thank you for listening to my TED talk ๐Ÿ™‚
u

user

09/02/2021, 3:05 AM
@Jeff Crooks sorry to ask you this but, could you transfer this to Github? At least we should you return about if possible/not implement the requested
u

user

09/02/2021, 10:42 AM
@[DEPRECATED] Marcos Marx Already did! #5765 , @s responded saying this is not only feasible but possibly implemented quickly! Any news on the Mongo source update?
u

user

09/02/2021, 12:43 PM
Also - I vaguely remember the ability to manually change the state? Is that a workaround I can use in the meantime?
j

Jeff Crooks

09/02/2021, 7:36 PM
yes, you could export the config and edit the job/attempts manually. I don't know if it's possible to do it now
3 Views