I had a chat with @John (Airbyte) and promised him to share the following two items
it would be nice if you would be able to start a pipeline with a certain selected docker container. In my case I need to set-up a VPN connection before extracting the data. I am now able to do this with the EC2 instance on which I am running Airbyte with docker-compose up. The docker containers make use of the underlying EC2 VPN connection, but I think a better model would be to build a docker container containing the VPN connection and then triggering a pipeline job with that specific docker container.
At a client of mine we are now running Dagter and we made use of the Dagster deployment to ECS approach, which worked quite well. The interesting thing is that the deployment to ECS makes use of a docker-compose file and starts the different services in different containers, connecting them as needed. I can imagine that this would be a nice approach for you guys as well, see the example deployment: https://github.com/dagster-io/dagster/tree/master/examples/deploy_ecs