Hi, I recently upgraded our MySQL connector to the...
# replication-troubleshooting
j
Hi, I recently upgraded our MySQL connector to the latest version (1.0.21). We had some huge issues with syncing after that (basically, everything stopped syncing), so we deleted all of our datasets in the destination, and deleted all of Airbyte's Connections, and created new Connections started from scratch. The initial syncs in the new Connections worked perfectly, but the issue that we're now running into is that every subsequent sync is doing the full dataset again, even though we are using the Incremental + Deduped strategy (which has always worked perfectly for us in the past). The logs show the following messages for every single Connection and Stream:
Copy code
No cursor field set in catalog but not present in state. Stream: [redacted], New Cursor Field: updated_at. Resetting cursor value
Any ideas on what is causing this, and how we can solve it? Thank you!
We are on airbyte version 0.40.18. Happy to upgrade if needed, but I don't think that's where the issue is coming from. Please correct me if I'm wrong
This is also interesting:
Copy code
airbyte=# select * from state where connection_id = '0b7ec27f-3b45-4c5b-be6d-0d503cef0e09';
-[ RECORD 1 ]-+----------------------------------------------------------------------------------------------------------------------------------------------------------------
id            | 08db6aae-276e-47bf-a017-2ae3f96bbc37
connection_id | 0b7ec27f-3b45-4c5b-be6d-0d503cef0e09
state         | {"cursor": "2023-01-31T10:04:59.000000Z", "stream_name": "foo", "cursor_field": ["updated_at"], "stream_namespace": "[redacted]", "cursor_record_count": 1}
created_at    | 2023-01-31 10:11:09.208619+00
updated_at    | 2023-01-31 10:11:09.208619+00
stream_name   | foo
namespace     | [redacted]
type          | STREAM
Why is the state only showing 1 stream_name, when there are 40 streams in that Connection?
n
Hi, Jonty! It's indeed always best to upgrade to the latest version if you can, just because so many changes are merged in all the time. I was about to say looks like the issue is with state and your latest message confirms it. How did you get these details, are you using the API? We have an endpoint that creates/updates the state of the connection, perhaps there would be a way to reset it with this: https://airbyte-public-api-docs.s3.us-east-2.amazonaws.com/rapidoc-api-docs.html#post-/v1/state/create_or_update
j
HI @Nataly Merezhuk (Airbyte) - I logged into the postgres db (hosted on docker)
n
Considering you deleted all destination datasets and connections to try and fix this, could you try a fresh install of Airbyte? I think you might be able to fix the state using the API, but it might take some experimentation. Have you been able to look into that?