Hey Team, My mailchimp connector keeps failing after trying to sync for 24h. Something went wrong during replication. Can someone have a look and help me please? I have attached the log file

Do you have only one Mailchimp connection? If you use a more recent date it works? Could you try another destination?

@Sebastian Brickel it looks like you’re using incremental append method for the sync mode. Have you tried the Full Refresh options? I’m running into a similar issue and I think I’ve only been able to load successfully when using full refresh.

The mailchimp connector unfortunately does not have a date window that would allow me to easily select only recent data. I also do not have access to another destination. I did figure out through trial and error that the email_activity seems to be the problem. However using full refresh is not really an option here considering the amount of data that we have. As you said @Samad Husain syncing for multiple days isn’t really sustainable

Agreed, I think we need to open a GH issue for this. In the meantime, I’m planning on running the Mailchimp syncs on a weekly chron schedule rather than a daily schedule and using the full refresh option if it continues to work.

Good point @Samad Husain. I will open a Github issue and might try the same that you are doing. I guess weekly updated data is still better than no data!

Try to increase the page_size paramter to 1000 maybe helps

@Sebastian Brickel are you using Airbyte Cloud? I’m using Cloud and I just ran into the 24 hour timeout that you originally mentioned with

Full refresh | Append

. Gonna try with

Full refresh | Overwrite

now.

@Samad Husain No I am using open source. @Marcos Marx (Airbyte) I tried with the page_size 1000, incremental deduped + history, which is way faster but still results in Failure.

Job either timed out or was cancelled.

Apparently I have data going back to 2020 in there that is trying to sync I have attached the log file I might try incremental append

Sebastian, can you check the state if it is updated? If yes, you can trigger a new sync to restart from where failed

I triggered a new sync soley running the email_activity stream and after 7h it already read 200000 records with over 1 GB, lets see if that works in the end or not. I will keep you posted @Marcos Marx (Airbyte)

The sync output any error? Would be nice create a Github issue to discuss with engineer about future solutions

I have uploaded the log file to the new Github issue I created for it. I have also closed the old one refering to the new one

Thanks Sebastian, this is impacting cloud users too. Hope to have updates any time soon