Airbyte missed a record while syncing from hubspot to postgres, I catched this error manually as there were 100 records only. How do I make sure this doesn’t happen at scale? Is there any way to catch these errors manually? My guess: Somewhere while requesting the data from hubspot, airbyte passed some timestamp (like last_synced) in payload. Hubspot didn’t return the data itself as the objects were created a few seconds apart only.

Hey @Abhishek Sachdeva, can you provide some example data with the timestamps? Would like to get a better idea of how airbyte is using the timestamp cursor. I’m not sure if there’s a way to catch these error automatically. My guess is you’d have to utilize a more fine cursor field than timestamp to capture the records that are missing. This would require a source code modification (I think)

Hey @Saj Dider (Airbyte) These were the timestamp of fields.

id                                               created_at
 3401 | "name83"  | "last83"                    | 2022-11-02 18:00:15.045+05:30
 3351 | "name82"  | "last82"                    | 2022-11-02 18:00:14.365+05:30
 3301 | "name81"  | "last81"                    | 2022-11-02 18:00:13.698+05:30
 2503 | "name80"  | "last80"                    | 2022-11-02 17:51:53.658+05:30
 3251 | "name79"  | "last79"                    | 2022-11-02 17:51:53.109+05:30

3301 was missing. First 80 were already present in db. When I added next 40, Airbyte should have fetched 81-120 but I got 82-120. I later found out the way to debug would have been to look at timestamp in airbyte’s db. Unfortunately, I refreshed everything before debugging.

I’m not sure if there’s a way to catch these error automatically.

is this a common occurrence?

Hello Abhishek Sachdeva, it's been a while without an update from us. Are you still having problems or did you find a solution?

Hey Marcos, I couldn’t replicate this bug. I’ll let you know if I face this again. Thanks for the support!