I’m running Airbyte using Helm on a Kubernetes cluster. In the
architecutral overview, we have the worker component. I noticed that this worker is always running, even when there aren’t any jobs. As I understand it, it is the worker that actually performs the syncs. I suspect that the worker is currently some monolith that operates based on the input from Temporal.
However, as I’m running in Kubernetes, I was expecting a worker that is only active when there is a job to perform. To this end I can dynamically scale up or down, depending on the load.
Q: Am I correct in my understanding of the system? If so, would it be possible in future that a worker is only active for the job that needs to be performed?