https://linen.dev logo
#feedback-and-requests
Title
# feedback-and-requests
b

Boopathy Raja

10/06/2021, 1:59 AM
VERY URGENT CRITICAL ⚠️ Hi, I was testing a mysql to Bigquery using CDC in airbyte. I put a table for CDC and configured it as Incremental model. When I ran the pipeline for the first time, it gets the snapshot of the table good. But, for the subsequent runs the new records where not captured. I used both Bigquery and bigquery-denormalised destinations. And the behaviour is same for both. Earlier when I was using the older version of airbyte, it was fine. But now, this issue is not solvable for me. Here is the log, in which the pipeline run successfuly, but failed to capture the new records.
u

user

10/06/2021, 2:31 AM
Copy code
2021-10-06 01:49:47 INFO () DefaultAirbyteStreamFactory(lambda$create$0):73 - 2021-10-06 01:49:47 [32mINFO[m i.a.i.s.m.DebeziumRecordPublisher(lambda$start$1):103 - {} - Debezium engine shutdown.
2021-10-06 01:49:47 INFO () DefaultAirbyteStreamFactory(lambda$create$0):73 - 2021-10-06 01:49:47 [32mINFO[m i.a.i.s.m.DebeziumRecordIterator(computeNext):95 - {} - no record found. polling again.
@Boopathy Raja when you run the first time did the initial load ingest all the data? and subsequent run didnt capture the new records?
z

Zach Brak

10/06/2021, 3:52 AM
@[DEPRECATED] Marcos Marx Exactly sir. In the initial run, it got all the rows got ingested and loaded. Then I inserted a new row in the source and ran the pipeline, but no rows got ingested.
u

user

10/06/2021, 9:22 PM
@Boopathy Raja what version of Airbyte and MySQL connector you were before starting the problem? Could you revert the mySQL to that version?
z

Zach Brak

10/07/2021, 5:52 AM
I haven't changed the version of mySQL 5.7, the airbyte alone I have upgraded from 0.27 to 0.29.2
u

user

10/08/2021, 1:21 AM
this is very weird @Boopathy Raja. Can you open a github issue to the team investigate further? Also, is it possible to share your mysql conf about the CDC params?
u

user

10/09/2021, 3:22 PM
Sure.. !
u

user

10/09/2021, 3:23 PM
I think there is a similar kind of issue already reported.. https://github.com/airbytehq/airbyte/issues/2599
u

user

10/10/2021, 2:12 AM
Can we reopen this issue? Or do I need to generate a new issue?
u

user

10/11/2021, 6:06 AM
Hey Boopathy can you create a new issue and share so that we can track this
2 Views