when reading records from a postgres source, does it do table by table or can or parallelize the number of tables being read?

Unfortunately, table by table, parallel streams are in the backlog: https://github.com/airbytehq/airbyte/issues/7749

any way to speed it up if i have like 200M million rows to ingest spread our across 100+ tables? Have compute to spare, saw something about allocating extra memory but that doesn't seem to be that large of a bottle neck

Best way right now is with incremental as far as I know. First load will be slow, but the following will be quick, as there will be far less records to read

@[DEPRECATED] Marcos Marx turned this thread into Zendesk ticket 2848 to ensure timely resolution!

Hey @Robert Put, Thanks for question. At the moment, it's not possible to parallelize the number of tables being read. You may want to consider deploying with Kubernetes. There are ways to increase job parallelism which may lead to faster syncs. More info here: https://docs.airbyte.com/deploying-airbyte/on-kubernetes/

Hey @Robert Put, Thanks for question. At the moment, it's not possible to parallelize the number of tables being read. You may want to consider deploying with Kubernetes. There are ways to increase job parallelism which may lead to faster syncs. More info here: https://docs.airbyte.com/deploying-airbyte/on-kubernetes/

the same options for more parallel works exist for the ec2 deployment, the issue isn't parallelization across multiple connections but in 1 connection. But like you said, thats not reall addressable right now without breaking the connection into multiple smaller ones

It’s unfortunate that we don’t have this feature at the moment. I’ve gone ahead and forwarded this to our product team. Thanks for raising this question.

i just asked in the help channel but is there anyway for a long initial sync to resume when it fails? or does it have to sync all the tables and restart if it failes? anything like that at the table level?

I opened an issue for that, but seems like it wasn't prioritized yet, if more people ask for it maybe Airbyte team speed that up. https://github.com/airbytehq/airbyte/issues/17774

Hey Robert, just saw your other thread so I'll respond there. Closing this for now. Let's continue the discussion there.

👍