Hi there! We are trying to implement Airbyte in our company right now. Our use case is some MySQL sources with a lot of events coming in and we are using CDC to replicate the changes on BigQuery. We create different connections from the sources (basically each table is a connection) to the destination. However we are getting a lot of failures related to bin logs not being available (Binlog mbin.017786 is not available. This is a critical error, it means that requested binlog is not present on mysql server). Some jobs complete with success after some attempts and others not. We have a huge retention policy already, so I don't think this is the problem. Looking to the logs the normalization process complete with success after the error but the state is still failed. Are we doing something wrong in our side? It's ok to have multiple connections to the same source running in parallel?

@[DEPRECATED] Marcos Marx turned this message into Zendesk ticket 2754 to ensure timely resolution!

Hi. In general it should be completely fine running few jobs on the same source table in parallel if you dont change the source. If you run parallel job on a destination table and one of the job changes it then this is a big no no, since the state of the table is being changed while other jobs are trying to query it - so in this case you need first finish the changing job and then run the other jobs. also, you dont need to do 1 connection for each table - you can do 1 connection for multiple tables if they are not related to each other - unless this is 100% crucial that you separate the 1 connection for 1 job which is also fine in some cases. if the job fails - would be great to get some insights such as logs - to see as to why it fails and how can we fix this

Hi there, Thank you for all the information! It's really helpful... I'm sending the logs of one case attached. The replication is running every 12 hours and it's we have a huge retention policy for the bin logs.

Hey could you create a discourse post so that team can look and get back to you

Hi there…. I’m afraid I don’t know what that is… 😅 It’s another slack channel?

Hey yeah you can create here https://discuss.airbyte.io