Hi guys! How are you?
I've tried to find any similar thread about the subject but I couldn't on this channel.
We have a lot of huge tables in PostgreSQL and I'd like to know if there is any way to speed up the ingestion or make a dump outside of Airbyte, load the data to data lake (S3) and after Airbyte just load data incrementally? I mean: first load would be done outside of Airbyte but incremental loads after that would be done by Airbyte.
🙏 1
Gleber Baptistella
02/08/2023, 12:51 PM
This thread seems to have the same situation, but without an answer... And the channel is archived.
d
Dave Tomkinson
02/08/2023, 1:46 PM
I thought i'd seen somewhere... but can't find where...
That someone had done a bulk load then gone into the airbyte db and updated the highwatermark manually in the tracking table.
Sorry can't find the link at the mo but it might give you something to search for... and I doubt it's best practice?
👍 1
u
user
02/08/2023, 1:47 PM
For that thread wasn't possible. Today you can load the data somehow in your final table using a dump, create the connection and edit the
state
table to use latest cursor field.
u
user
02/08/2023, 1:47 PM
There isn't instructions how to do it today, I'll add it to my list for. afuture tutorial :D