Hi all, I am just about to get started with Airbyte and my expected deployment is on an EC2 VM. I just have a question with respect to how and where the data is moving from source to destination. My source is likely an AWS s3 bucket or Amazon RDS living in a VPC and the destination is Snowflake setup with AWS PrivateLink to our source. If we have our EC2 VM deployed on the same AWS network will Airbyte be moving the data across the public internet or will it leverage the PrivateLink? Due to security risks we don't want this data in particualr to move across the public internet

hey Kyle, as long as they are deployed in the same network and you supply the snowflake-provided privatelink host address it should not be going over the public internet. Airbyte would just use whatever DNS resolution is available on the host node, which in the case of your setup should just point to the snowflake host inside the VPC is network egress enabled in this VPC? one way to be super sure would be to turn it off on the Airbyte node (potentially via security groups) and run a connection

amazing, thank you!!! We have yet to set this up yet but will likely do so this week. This is extremely helpful

yup, should be the place

one last thing, is there support for Lambda? and how soon is ECS coming per the docs?

to be 100% ECS is not on the short term roadmap, maybe in 3-6 months, but that’s really not firm. Can you upvote the ECS issue on github to help us gauge interest/value?

to deploy on lambda instead of EC2?

oh I see, deploying airbyte itself on lambda is probably not in the roadmap for the forseeable future due to timeouts, memory limits, and costs of long-running jobs on lambda