Hey everyone, We are evaluating Airbyte and have ...
# announcements
v
Hey everyone, We are evaluating Airbyte and have a few questions : 1. How does the current Airbyte RDBMS source connectors deal with loading of very large source tables (Table sizes in 100s of GBs ). Any splits based approach? 2. Any support for custom queries (support for filters, select specific columns, functions, etc) in the RDBMS source connectors. 3. Any plans for supporting cron tab expressions in scheduling?
1
d
Hi Vaibhav, 1. Inserts speeds are destination dependent. Our current RDBMS all use inserts and aren’t the most efficient. I’m currently overhauling our Redshift destination to use upload to cloud storage + copy which is an order of magnitude faster. We have a ticket to improve this, and it’s a matter of prioritisation. So let us know a specific warehouse is blocking you. 2. We do not support selecting a subset of columns when replicating a table. Is this what you were thinking of? 3. Yes. We have considered this idea and have this ticket for tracking, however this honestly isn’t high on our list. Feel free to post on the ticket! Does this make sense?
c
For your point number 2, we have the following issues if you want to jump in the discussion and add some comments about your use case: • https://github.com/airbytehq/airbyte/issues/2463#issuecomment-799630461https://github.com/airbytehq/airbyte/issues/2227#issuecomment-799646008
On the first point, there is also this thread for another alternative to split large sources: https://airbytehq.slack.com/archives/C01AHCDHCKE/p1616511289036600?thread_ts=1616410330.028300&cid=C01AHCDHCKE It’s related but a different way of handling large sources compared to using the batch cloud storage approach that Davin linked to