Hello all, is there any possibility of supporting a “split-worker” mode where an Airbyte connection launches multiple workers, with each worker being responsible for a certain subcomponent of the source, e.g. one worker per repository for a Github source?

@Christopher Wu the idea is to parallelized the process of pulling data?

Yes. I realize this parallelization could also be done inside the source itself.

@Christopher Wu would you be open to creating an issue for tracking this?

I can do that

@Christopher Wu Is this driven by concern that one node won't be fast enough for a single source ?

That was one concern for our use case, but that was mainly driven by strict aws lambda execution timeouts, which airbyte does not have. Another is that some entities of a source, e.g. individual projects from a Jira source, have insufficient permission configurations or other misconfigurations that we don’t want to our jira connection to fail on, i.e. the connection should log the error and keep running as normal.