https://linen.dev logo
#feedback-and-requests
Title
# feedback-and-requests
s

Sadik Bakiu

07/20/2021, 5:22 PM
Hello all, is there any possibility of supporting a “split-worker” mode where an Airbyte connection launches multiple workers, with each worker being responsible for a certain subcomponent of the source, e.g. one worker per repository for a Github source?
u

user

07/20/2021, 5:48 PM
@Christopher Wu the idea is to parallelized the process of pulling data?
u

user

07/20/2021, 5:49 PM
Yes. I realize this parallelization could also be done inside the source itself.
c

Christopher Wu

07/20/2021, 6:10 PM
@Christopher Wu would you be open to creating an issue for tracking this?
u

user

07/20/2021, 6:29 PM
I can do that
u

user

08/08/2021, 9:08 AM
@Christopher Wu Is this driven by concern that one node won't be fast enough for a single source ?
u

user

08/09/2021, 5:49 PM
That was one concern for our use case, but that was mainly driven by strict aws lambda execution timeouts, which airbyte does not have. Another is that some entities of a source, e.g. individual projects from a Jira source, have insufficient permission configurations or other misconfigurations that we don’t want to our jira connection to fail on, i.e. the connection should log the error and keep running as normal.
4 Views