Airbyte is an open-source data integration engine that helps you consolidate your data in your data warehouses, lakes and databases.

Airbyte

@here We are syncing events from kafka to redshift. We have deployed the airbyte on kubernetes . Sometimes our pipeline is getting stuck. Source and destination pods are in runniing state for large amount of time. I have seen the instances where our destination redshift pod is running for 3-4 hours.

Yes the data is in range of 500Mb to 1 gb.

Memory consumption is
```destination-redshift-sync-10956-0-ikrut   8m           137Mi
destination-redshift-sync-10961-0-tsihn   8m           137Mi
destination-redshift-sync-10972-0-ocsyy   55m          413Mi
destination-redshift-sync-5102-1-zxfio    0m           1Mi
source-kafka-sync-10972-0-iloko           26m          2011Mi```

<@U02JJ554F2T> How can we increase the memory and cpu allocated to source and destination sync pods

Hey in the latest builds we have added the support over API for connections you can check it <https://airbyte-public-api-docs.s3.us-east-2.amazonaws.com/rapidoc-api-docs.html#post-/v1/connections/create>