Hello guys! I'm using MWAA on a project and I was wondering, is there a way to integrate Airbyte into my solution without using external compute, like EC2, or ECS?

Airbyte isn’t currently set up to run in ECS but it should run fine on EC2. Is that enough to answer your question, or do you need something more specific?

Hi @Wes T, we’re also looking to orchestrate Airbyte with MWAA. After failing to make it work, we decided to go with Meltano. I asked for guidance in #airflow but haven’t received any feedback yet. Nonetheless, I’m keeping an eye on Airbyte’s progress with Airflow. Hopefully, there’s a more direct approach soon.

I have some ideas for trying to get airbyte going on ECS, but I don’t know if they’ll work. They do require some changes to the core airbyte software, to accomplish it on ECS. I’m on more pressing features at the moment or I would take a deeper look. It’s on our radar, though, along with other kinds of deployment recipes to make everyone’s lives easier for deployment.

So, the idea was to get it running on MWAA, not on ECS, neither ec2. That's a serverless service from amazon.

The tricky part is how the docker containers communicate with each other, and what that implies for what runs on the airflow worker node. I can’t remember whether MWAA supports sidecars or if you only get one docker image for a worker node. Airbyte has the web app (single node), the scheduler (single node), and the workers (some scale-out capabilities). I presume you’d be trying to integrate an airbyte worker’s docker image contents with an airflow worker’s docker image contents and then call the worker from airflow on the same running docker container? Just trying to understand your use case goal.