https://linen.dev logo
Join Slack
Powered by
I am wanting to test out AirByte and we are requir...
# replication-ask-ai
y
I am wanting to test out AirByte and we are required to use ECS. I found this https://docs.airbyte.com/deploying-airbyte/on-aws-ecs/ 
On AWS ECS (Coming Soon)
How soon is that comming?
k
I'm sorry, but based on the information provided, there is no specific timeline mentioned for when Airbyte will support deployment on AWS ECS. The current version of Airbyte is not compatible with ECS as it relies on Docker containers being able to create other Docker containers, which ECS does not permit. However, it is mentioned that they will be revising this strategy soon to be compatible with ECS and other container services [source](https://docs.airbyte.com/deploying-airbyte/on-aws-ecs). I recommend keeping an eye on the official Airbyte documentation and their product roadmap for updates on this feature.
y
But this ticket seems to indicate it may work but can't find any details on how to set it up: https://github.com/airbytehq/airbyte/issues/4847#issuecomment-1533276511
c
It might work, but it’s not actively supported by us currently
If you are looking for general guidance I can try to assist you. The problem with ECS is that it does not support container orchestration as a first class citizen (e.g. docker-compose like setups). My understanding is that there has been recent work in this space to make it easier, but it’s not something we have actively looked into yet
looking into this briefly a bit more, supposedly AWS “Batch” allows dynamic creation of containers in ECS. There would probably have to be some application level work to make this viable in Airbyte: https://stackoverflow.com/questions/66349249/ecs-fargate-is-it-possible-to-create-container-instances-dynamically
y
@Conor Barber (Airbyte) Thank you. I will give it a try but I am generally scared of "unsupported" features. Sounds like this is something we should not trust in our production environment.
c
given that there is no builtin way to dynamically spawn containers in ec2 or really to network them together without some application changes I am skeptical that this will work out of the box at all
e.g. it’s probably not even worth trying
y
thanks.
does 
On AWS ECS (Coming Soon)
really mean 
On AWS ECS (Coming No Time Soon)
c
I think that’s a fair bet, because it’s not on our immediate roadmap
I guess that you could call it Blizzard Soon (tm)
🫠 1
however if you make some noise about it that could cause us to reprioritize 😄
y
ok, I can try 🙂
@Conor Barber (Airbyte) Sorry, I am back after doing a little thinking about this. I am trying to understand the problem better. We run Dagster, and it spawns other docker instances (workers) in ECS. We use SecurityGroups to control routing there. I am trying to understand how AirByte's use-cases are different from Dagster in this case and more importantly why this model would not also solve AirByte deploy in ECS. (Sorry, I know very little about AirByte which is why I want to test it out)
c
So it is possible to spawn containers in ecs. The problem is that Airbyte is the one spawning jobs containers. Until Airbyte has a way to explicitly spawn these containers in ecs (the only supported orchestration methods that Airbyte has right now are via Docker Compose and Kubernetes), it will not work, since ECS does not have native support for docker-compose
so I guess to answer your question, yes, this model would probably allow you to deploy airbyte in ecs that you mentioned, but it would require application level change to support it in airbyte (probably in the code that handles job scheduling, adding a new scheduler or similar for ecs). Alternatively, if ecs supported docker compose for scheduling and orchestration, it would probably just work out of the box
y
So, I think I have a way forward. So I think the answer is AirByte will run just fine in ECS but you have to use EC2 and not Fargate. I am going to following this guide but have Docker do all the steps in stead of CLI.
So, this will "spawn" containers on the EC2 instance but won't bring up addition "severs". They will all exist on a single host... Maybe the "does not work on ECS" mean it can't spawn additional ECS instances.
This is for sure not ideal but I think it "should" work fine?
c
yeah, so you would have to use a fat host and could not scale horizontally
👍🏼 1
but if that works for your use case that would be fine
y
I guess I am not sure if it works for my use case or not... But will give it a try..
c
I will say airbyte is not THAT heavy lifting, but if you are trying to run a bunch of syncs concurrently you might run into problems as it’s going to spawn 3-4 pods per sync
👍🏼 1
and those pods are going to ask for some defaults that are relatively high (i don’t know what off the top of my head, but several gb’s ram and some cpu
y
How does this article expect scale to happen in production? https://docs.airbyte.com/deploying-airbyte/on-aws-ec2/
c
that all can be configured, I think airbyte’s defaults are kind of greedy and depending on what you are doing the sync pods can probably be significantly scaled down
y
Looks like it just indicated t2.large
c
yeah, I would start with that. I am not sure how many concurrent syncs that size recommendation came from, but you can always start there and make it bigger if you need to
y
ok, sounds like this at least lets me get my feet wet and have a starting point with AirByte
c
a couple of hints if you do run into size constraints, stagger your syncs so that they don’t run concurrently, and experment with downscaling the size of the sync pods. They are aggressively defaulted for stability reasons, but because everyone’s data is different oftentimes they don’t need near the space that the defaults indicate
👍🏼 1
in other words, one person’s memory requirements for a postgres source may be radically different than another person’s simply because the input data is very heterogenous
y
understood
Open in Slack
PreviousNext
Powered by