Airbyte is an open-source data integration engine that helps you consolidate your data in your data warehouses, lakes and databases.

Airbyte

Hi Airbyte team. One feature suggestion. We have a somewhat large Airflow deployment (1000+ pipelines) and we leverage temporary workers in the cloud. Can Airbyte also use Kubernetes Pod as one off worker to execute tasks?

Airbyte can be deployed in Kubernetes, but are you asking if Airbyte can be deployed in a service like Amazon ECS?

Can Airbyte not having fixed workers standby and only schedule worker pod when required?

Hey we have workers as fixed but the sync pods (source, destination) get created runtime and exited runtime.

Is there something more you are hoping for ?