Hello Team, we are starting with Aribyte and we are stuck about parallelism. We deployed Airbyte in Kubernetes and following this doc we added the those variables in the worker deployment, despite of this, we still have one pod for source and one pod for destination and the data transfer is taking always the same time there is not improvement on that. Our question is: How is parallelism handle by Airbyte? should we have more pods for source/destination?

what source? for example postgres has no parallelism across all the tables syncing. its 1 table at a time for all the tables in the job https://github.com/airbytehq/airbyte/issues/7749 • is a request for it and its epic: https://github.com/airbytehq/airbyte/issues/7750 those values would help run multiple jobs at once, but not speed up the job itself for 1 source

Hello Robert, thanks for your answer, the source is MSSQL.

so yeah, the only option would be to split it out to multiple jobs yourself

Ok. Thanks, Lets do that.