Hello, I am using incremental sync + deduped hist...
# replication-troubleshooting
a
Hello, I am using incremental sync + deduped history for a couple of tables syncing from MySQL to Redshift. Now, syncing it for the first time results in the right amount of entries. But syncing it a second time OR syncing any other table will result in almost all entries being duplicated. I can overwrite the data by doing a Full Refresh | Overwrite but changing it back to incremental sync and syncing a couple times will result in ALL my incremental final tables having duplicate entries. The scd table will contain the two different records from syncing twice, with marking one of the records correctly as __airbyte__active_row = 1, but the final table will have the record with __airbyte__active_row = 1 twice. The _airbyte_unique_key column is NULL for one of the two records in the final table. Any ideas? Thanks. airbyte version 0.40.17, all connectors updated to latest version.
n
Hi Annika! First, make sure to update Airbyte to the latest version. Looks like the cursor may be set incorrectly, what are you setting the cursor field to?
🤔 1
a
The cursor field is an updated_at column or modified_at column.
n
Could you please give me bit more info: 1. Airbyte version 2. Connector versions 3. Any sync logs
a
We updated Airbyte and it seems like so far we don't have the problem anymore. Now I am running into an issue with the Zendesk connector though (airbyte version 0.40.32, zendesk connector version 0.2.21). It won't reset and sync the tickets table unless I turn it off. I have it on Full Refresh | Overwrite write now and the rest doesn't succeed. Here are the logs. It works for all other zendesk data.
Copy code
2023-02-07 15:51:56 [42mnormalization[0m > KeyError: '$ref'
2023-02-07 15:51:56 [32mINFO[m i.a.w.n.DefaultNormalizationRunner(close):190 - Terminating normalization process...
2023-02-07 15:51:56 [1;31mERROR[m i.a.w.g.DefaultNormalizationWorker(run):83 - Normalization failed for job 301.
Copy code
io.airbyte.workers.exception.WorkerException: Normalization process did not terminate normally (exit code: 1)
	at io.airbyte.workers.normalization.DefaultNormalizationRunner.close(DefaultNormalizationRunner.java:201) ~[io.airbyte-airbyte-commons-worker-0.40.32.jar:?]
	at io.airbyte.workers.general.DefaultNormalizationWorker.run(DefaultNormalizationWorker.java:81) ~[io.airbyte-airbyte-commons-worker-0.40.32.jar:?]
	at io.airbyte.workers.general.DefaultNormalizationWorker.run(DefaultNormalizationWorker.java:34) ~[io.airbyte-airbyte-commons-worker-0.40.32.jar:?]
	at io.airbyte.workers.temporal.TemporalAttemptExecution.lambda$getWorkerThread$5(TemporalAttemptExecution.java:195) ~[io.airbyte-airbyte-workers-0.40.32.jar:?]
	at java.lang.Thread.run(Thread.java:1589) ~[?:?]
2023-02-07 15:52:04 [42mnormalization[0m >   File "/usr/local/lib/python3.9/site-packages/normalization/transform_catalog/transform.py", line 104, in main
2023-02-07 15:52:04 [42mnormalization[0m >     TransformCatalog().run(args)
2023-02-07 15:52:04 [32mINFO[m i.a.w.g.DefaultNormalizationWorker(run):94 - Normalization executed in 2 seconds for job 301.
n
Could you please include a full log file?
a
@Nataly Merezhuk (Airbyte) I got an answer in a separate thread, thank you.