Hi folks, How complicated it is to implement an executor for Airbyte to allow running connectors on AWS Lambda (instead of EKS)? (I am asking, because we are using AWS Lambda as our main runtime and it’s been a breeze)

This is an interesting idea. I don’t think we have evaluated this implementation before. But theoretically it is possible. If this is an important feature for you, would you mind creating an issue on GitHub for us to track and prioritize it in the future? https://github.com/airbytehq/airbyte/issues/new?assignees=&labels=type%2Fenhancement&template=feature-request.md&title=

According to this doc , the Function timeout is 900 seconds (15 minutes) which would be a big problem for jobs.

Won’t it be expensive to use lambdas for such use case? I believe that it suits the best for more often but faster executions. I suppose it won’t fit for full-refresh approach but it could work for incremental sync. Also probably step functions could be used for long-running computations but needs more complex logic + the price will be much higher than using just eks.

Based on our usage AWS Lambdas are very cheap compared to on-demand/spot compute instances.