https://linen.dev logo
Join Slack
Powered by
Hi, Airbyte says successful sync (incremental ref...
# announcements
d
Hi, Airbyte says successful sync (incremental refresh - based on 
id
), but I am checking the table in the destination db, it has exactly the same number of rows as before, even though Airbyte says it synced ex. 17,009 records, and I see it still has the old count 16,539 Any chance I can browse logs more deeply?
c
Screen Shot 2021-01-20 at 1.21.20 PM.png
to get logs for a given sync there's a download button that will grab all the logs for that sync.
🙏 1
i'm happy to try to help as well.
what is the name of the table you are looking at in the destination.
d
it is a short log, I don’t see anything special in it. The sync says 
Succeeded
, 
11.84MB | 31,793 records | 23s
But in the target database I still se 
31,474
records as before
log.txt
c
(I want to make sure you're not looking at a temporary table or something)
d
it is a table called 
events
, and I did a db_dump in the target database, so I have the exact data model/schema
I mean, I did a db_dump of an older snapshot (with less records)
c
want to make sure i understand the order of events: • you manually dumped your table into your target database (an older version with fewer records) • then you ran airbyte on a newer version of the table
yeah?
if every thing i said above is accurate, then the issue is probably that airbyte is not writing data into the table you think it is.
i can probably help you find the table airbyte is writing if you can tell me: source type (e.g. postgres), table name, destination type (e.g. postgres, bigquery)
d
Source database is RDS Aurora Postgres v11.9, and the target is also RDS Aurora Postgres v11.7
I created a 
pg_dump
a month ago from the source database, I loaded that dump into the destination database using 
psql
. Database schema and model are the same, the only difference is source database has more records than the destination database
c
you are doing everything in the default postgres schema? 
public
if so then the table that airbyte is writing to in your destination is 
"public.events"
is that the table you are looking at or are you just looking at 
events
d
I have only one schema (
public
) in both source and destination databases, so 
events
is always public.events
c
ah. okay. sorry what i said wasn't clear.
the table you dumped is called 
public.events
. the table airbyte is writing to is called 
public."public.events"
.
we include the source schema name in the table name in the destination.
d
do I look the name in the 
Cleaned name
column?
c
unfortunately no. we're working on getting those to line up, but they don't right now.
d
image.png
c
do you have a postgres console open for your destination?
d
yes
c
if you run 
\d
it should show you all the tables
hopefully we can see it in there.
d
that is the only events table I see
c
for your configuration of your postgres source in the UI, is the basic normalization toggle set to true?
d
Is it turned off by default? I think I didn’t turn it on
c
it is off by default.
(for now) we actually plan to change this in the next day.
d
should I turn it on?
c
yeah. i think turning it on will get the behavior you're expecting.
did that work out okay for you?
d
ok, now it works
c
woohoo!
d
but how do I map 
public.events
from source to 
public.events
in destination?
c
unfortunately airbyte can't do that for you right now.
d
do you plan to support that?
c
yes.
you can achieve effectively the same thing by using a postgres view.
CREATE VIEW events AS SELECT * FROM public."public.events";
d
yes, but I need to send 40gigs of data either way over the network?
c
so you'll be able to query the airbyte version of the table as if it were just name of the table you expect.
using a view does not change how much data you need to transfer at all.
it will be referring to the airbyte version of the table in the same db.
d
let’s say source table events is 40gig (the whole database is actually 40gig, but for the conversation let’s assume that table is 40gig). That means that Airbyte will transfer 40gig of data from source to destination
c
yes.
d
there is not a way for me to load a snapshot which will create the tables, and then use Airbyte only for the incremetal refresh? What if I load a snapshot in destination and rename the table to public.“public_events”?
c
ah. interesting.
it seems theoretically possible but i have not tried it.
actually.
i take it back, it's not realisitically possible right now to pre-seed the destination table.
d
Because our databases are VPC peered over a Transit Gateway, imagine if our source db is a pentabyte, moving that kind of data will incur nice AWS costs due to the traffic
c
ouch. yeah.
the issue is that airbyte handles the state of an incremental sync internally and doesn't really provide a way for you to tell it to start from not the beginning.
so the first sync will be expensive (all 40 gigs)
but then after that it will only send the new data.
i'm not too familiar with the Transit Gateway... by manually uploading the SQL dump is that cheaper than sending the data through the Transit Gateway? or does it incur the same cost?
d
it is not that much, $0.02 per GB sent over Transit Gateway (TG is a service to centralize VPC peering), so that is not a problem
What are the performance caps/metrics for Airbyte? Can we scale it?
c
currently each sync is single threaded. for relational databases that should generally be fine.
we are planning to add parallel processing support for larger scales, but again, that's generally going to be overkill for a relational db.
d
ok, so it should handle if I have a DB with around 90 tables and 45gig of data?
c
yeah. that seems well within the bounds of what we expect to support.
d
running for 20m, I’ll wait it out for an hour more, hopefully it should finish by then. But I am looking at the metrics at my DB, I don’t see any special count of write operations
c
I don’t see any special count of write operations
sorry, what does this mean?
d
write operations, they are quite low and database CPU usage is at 1%. If it breaks for some reason, it should log? Does it have some sort of auto-retry?
c
yes. it does have auto retry.
write operations, they are quite low and database CPU usage is at 1%.
hahah this is sort of a good thing. means we're not murdering your db 😄
it writes in batches to the database so that it doesn't hurt it too badly, so i'm not too surprise.
i'd expect that the bottleneck is network more than it is writing to the database.
d
ok, so 90 tables, 45gb of data, it writes in batches, give or take, ~2h?
c
yeah i would hope so. on a purely local setup, airbyte reads about a gig of data in a little less than a minute. include network and having to write as well, and it's likely to be more than so 2 hours doesn't seem crazy.
5 Views
Open in Slack
PreviousNext
Powered by