I’m running Airbyte using Helm on a Kubernetes cluster. In the architecutral overview, we have the worker component. I noticed that this worker is always running, even when there aren’t any jobs. As I understand it, it is the worker that actually performs the syncs. I suspect that the worker is currently some monolith that operates based on the input from Temporal. However, as I’m running in Kubernetes, I was expecting a worker that is only active when there is a job to perform. To this end I can dynamically scale up or down, depending on the load. Q: Am I correct in my understanding of the system? If so, would it be possible in future that a worker is only active for the job that needs to be performed?

The worker is responsible to launch the source and destination, so you need to keep at least one container running to start the job.