https://linen.dev logo
Join Slack
Powered by
Is it possible to replace the (godawful) airbyte r...
# announcements
c
Is it possible to replace the (godawful) airbyte redshift target with one from singer or pipelinewise?
✅ 1
m
the speed is the problem?
c
oh it’s awful.
it’s not using s3 loads for starters
a simple accounts table only load from SF so far is taking hours
it’s like it’s just using the postgres driver raw
so far meltano is about 50x faster
perhaps 100x
m
Got it! let's fix the problem, we are actually going to rework a part of the ingestion to support loading from files because we need it for delta-lake
c
does airbyte not use the singer targets directly then?
m
it depends which sources and destinations
c
can we switch the redshift target?
m
but to be clear the reason pipelinewise is so fast is because they don't use singer, they use their own "fastsync" pipeline
c
oh not pipelinewise
even the native singer target is 100x faster
m
Thinking how we can prioritize this fast
I hate that we are so slow
c
you need to load from s3, with one file in s3 per node in the redshift cluster
m
yep
c
also not to use json blobs
because redshift doesn’t have native support (yet)
m
that's OK. we can work with json blob after loading
even if it is not supported
c
yeah, but it will be sloooooow
m
is really just the loading part
why do you think so?
c
because it has to string parse it, there’s no native json type
you should load the json to s3
then s3 straight to tables and columns in redshift
m
how do you deal with nested models right now if you use this method?
c
so far haven’t used one tbh, don’t know
m
you don't see the problem with Salesforce right now?
c
no metlano can load it in 20 mins
so far I’ve been watching the airbytes load table and it’s adding like a few hundred rows an hour, of json
m
but what does it do with nested models?
c
hasn’t done any postprocessing yet
salesforce doesn’t have any nesting that I’m aware of
m
ah
d
To get sense of scale, how big is the table you are loading Chris?
m
I added our conversation there
👍 1
c
80,000 rows
m
Thanks chris. Do you have a github handle?
👍 1
c
chriskl
m
I'll tag you on the ticket so you can follow the progress. we need to address it quick.
j
@Chris Kings-Lynne, as for JSON processing in Redshift, doesn't Redshift have built-in functions for parsing JSON? We're using Exasol, which also doesn't have a native JSON type -- JSON is just stored in varchar columns. However, I can transform 150 million rows of JSON data into columns in a new table using JSON_EXTRACT() in just a couple of minutes on our 4-node EC2 cluster (m5.16xlarge).
Most of our pipelines are definitely not at that scale (the above is for the front-end tracking data), but rather maybe around 100k - a few million rows daily, so JSON processing performance hasn't been an issue at all so far.
👍 1
s
Just FYI to everyone here — we just released a new version of the Redshift destination connector that uses 
COPY
to move data. It is lightyears faster and should solve this issue. Thanks for your patience everyone! We’ll also be releasing the same strategy to all compatible warehouses in the near future.
Open in Slack
PreviousNext
Powered by