hello guys, how do you implement connectors for large datasets? I am having a hard time iterating through all of the records of a dataset with more than a few million records

In the same situation. Ingesting hundreds of millions of records from S3 to Redshift, and eventually API to redshift and it keeps just dying after a couple million.

I guess this issue that I opened yesterday is related to your question https://github.com/airbytehq/airbyte/issues/19572

I am writing a custom connector @Ethan Brouwer but I guess the problem is similar in a way. The large amount of records forces the syncing to be way too slow when it doesn't kill the process. @Murat Cetink it is related but I must collect the whole dataset, the job I am creating must collect open information from a website and save it to a cloud.

Hello Gustavo Maia, it's been a while without an update from us. Are you still having problems or did you find a solution?