Hi folks anyone seen this error before?

java.util.concurrent.ExecutionException: io.airbyte.workers.general.DefaultReplicationWorker$SourceException: Source cannot be stopped!

i get this error every time i try to sync something bigger then 1-2 million rows and i cant really find what is going on . I am doing a Postgres -> MSSQL connection , its pretty simple , its 4 tables , and i want them to be dedup incremental . the 2 smaller table, works like a charm the bigger ones always get this error even if i try different modes or combinations of tables , and all tables get reset everytime it happens 😞

Hey @Nicolas, the source cannot be stopped error is a fairly generic one and it’s a bit difficult to trace what the issue could be on a first glance. Large syncs are generally more fragile and we’re ceaselessly trying to improve performance. A common issue with folks who get this error is that they underestimate how many resources they need to allocate and that causes some of the workers to eventually fail (which I know is frustrating for large syncs). How have you deployed AIrbyte and how many resources have you allocated for the instance? You can read more about scaling airbyte here; https://airbytehq.github.io/operator-guides/scaling-airbyte/

Hello @Sajarin Dider thanks so much for your reply, i used the docker deployment on azure , i followed the documentation and didnt do any customization. The vm i chose is a 8vcpu 64gb ram which seemed sufficient for the task ; in data terms , my large tables are not THAT big; the largest one has 3.4gb in 9.7 million rows i understand the row size is below the limitations mentioned in documentation is there any additional setup i might be missing out? I have 0 expertise in airbyte i am just attempting to fit airbyte in any of my current flows atm 😞

Hello Nicolas, it's been a while without an update from us. Are you still having problems or did you find a solution?

@Marcos Marx (Airbyte) thanks for returning, unfortunately i didnt fix my problem; the overall: From my attempts i noticed i get the “Source cannot be stopped” whenever a sync task takes over 1 hour, so basically everything with more then ~1 million rows ends with this error, if i keep retrying the sync process it eventually manages to finish as it has middle checkpoints but not really effective and observability goes to down to 0 . i am assuming what i want to do is to simple for what airbyte is ment to do, hence i am not getting the results i am expecting, even when i finally manage to complete the sync, the normalization phase takes ages to complete , even on incremental updates , one of the tables takes 2-3 hours everytime it has incremented lines even when are as few as 200 lines , i am assuming the way Airbyte deals with this processes are ment for way more complex scenarios then mine so the results turns out to be below expected for my scenario , as 2 hours to sync 2.3k lines is not what i was expecting. still a good solution though, probably just not the one i need thanks a bunch for returning

hey, if you are having this issue, the solution for me was to increase resources for the sync workers. You can do that by changing the limits of the container inside Airbyte DB, this doc helped me to do that: https://docs.airbyte.com/operator-guides/configuring-connector-resources/#configuring-connector-specific-requirements

Hey Nicolas, were you able to get this working? Does Gabriel's suggestion help? (Thanks Gabriel!)