Joël Luijmes

01/17/2022, 11:10 AM
I’m running Airbyte using Helm on a Kubernetes cluster. In the architecutral overview, we have the worker component. I noticed that this worker is always running, even when there aren’t any jobs. As I understand it, it is the worker that actually performs the syncs. I suspect that the worker is currently some monolith that operates based on the input from Temporal. However, as I’m running in Kubernetes, I was expecting a worker that is only active when there is a job to perform. To this end I can dynamically scale up or down, depending on the load. Q: Am I correct in my understanding of the system? If so, would it be possible in future that a worker is only active for the job that needs to be performed?

[DEPRECATED] Marcos Marx

01/18/2022, 12:06 AM
The worker is responsible to launch the source and destination, so you need to keep at least one container running to start the job.